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The  Air  Force  Human  Resources  Laboratory  began  development,  of  the  Civilian 
Potential  Appraisal  System  (CPAS)  In  July  1977  at  the  request  of  tlieAlr  Force 
Directorate  of  Civilian  Personnel.  The  system  was  to  provide  a  rating  and 
promotion  eligibility  ranking  process  for  all  non-senior  executive 
appropriated  fund  employees.  The  system  was  to  be  part  of  the  final  ranking 
process  In  the  Air  Force  Promotions  and  Placements  Referral  System  (PPRS). 
The  primary  objectives  of  the  CPAS  were  to  restore  credibility  to  the  civilian 
promotion  appraisal  process,  to  be  operationally  efficient,  and  to  deter 
rating  Inflation. 

The  development  process  established  a  large  pool  of  possible  rating 
factors,  produced  prototype  experimental  data  collection  devices,  organized 
the  large  number  of  civilian  specialties  Into  a  manageable;  number  of 
homogeneous  job  families,  collected  experimental  rating  data  on  a  latge  sample 
of  cases,  and  reduced  the  rating  factor  pool  to  a  workable  set.  The  set  of 
ratings  was  then  presented  to  job-family-speelflc  panels  of  subject-matter 
experts  who  selected  rating  factors  appropriate  to  their  job  family  and  sat  as 
•a  promotion  board,  ranking  employees  according  to  promotablllty.  The  panel 
ranking  processes  were  reproduced  mathematically  by  judgment  analysis; 
techniques  providing  formulas  or  algorithms  for  operationally,  weighting  rating 
factors  for  use  In  the  CPAS. 

Legal  defenslblllty  of  the  promotion  appraisal  system  was  a  major  goal  of 
the  project.  Under  the  1978  Uniform  Guidelines  for  Employee  Selection,,  any 
system  which  results  In  adverse  Impact  on  minority  groups'  must  have  fulfilled 
several  developmental  requirements  to  establish  job  relatedness  and  the 
validity  of  the  system;  otherwise,  the  user  would  be  liable  under  the 
guidelines.  The  probability  that  any  major  selection  system  will  have  adverse 
Impact  Is  sufficiently  high  that  all  such  systems  should'  be  developed  adhering 
to  the  guidelines.  To  satisfy  the  guidelines;  all  possible  selection  system 
components  must  be  considered  and  the  final  set'  systematically  selected  to 
provide  a  system  which  maximizes  objectivity  and  validity  while  minimizing  the 
number  of  elements  that  contribute  to  adverse  Impact.  Elements  showing  the 
least  amount  of  bias  should  be  selected,  but  hot  to  the  point  of  sacrificing 
validity.  That  Is,  selection  system  components  must  be  valid,  and  .whan  a 
choice  must  be  made  among  candidate  components  which  have  equal  validities, 
those  with  the  least  potential  for  adverse  Impact  should  be  chosen.  However, 
when  the  validities  are  not  equal,  components  wl.th  the  higher  validities 
should  be  chosen.  Where  criterion  validity  Is  not  available,  construct  and 
content  validity  should  be  used  to  develop  the  system,  and  criterion 
validation  should  become  an  ongoing  process  during  system  use.  CPAS  was 
designed  to  meet  these  requirements. 

Three  categories  of  Information  were  considered  for  Inclusion  In  CPAS; 
demographic,  performance,  and  aptltudlnal.  To  obtain  these  data,  the 
Demographic  Questionnaire-Worker  booklet,  Job/Worker  Characteristics  Ratings 
booklet,  and  Civilian  Personnel  Examination  and  Score  Checking  Test  were 
developed.  ' 


i 
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Tft*  approximately  1,500 civilian  job  series  were  collapsed  by  two  teams  of 
Industrial  psychologists  and  expert  position  classifiers  Into  23  job 
famillesi  The  job  series  were  clustered  Into  homogeneous  groups  according  to 
similarities  In  job  tasks'  and  skills,  and  the  clustering  process  was 
cfoss-vaiidatid. 

The  data  collection  instruments  were  twice  field  tested,  revised,  and  then 
disseminated  to  a  representative  sample  of  20,000  civilian  job  Incumbents  and 
their  supervisors.  The  CONUS-wlde  data  collection  effort  resulted  In  12,865 
or  64;3S  complete  usable  data  sets. 

A  variable  reduction  process  evaluated  each  candidate  variable  (Items  from 
thb  data  collection  Instruments)  on  a  number  of  criteria  and  reduced  the  Set 
to  a  manageable  number  of  24  variables  for  further  consideration.  The 
criteria  used  to  reduce  the  overall  number  of  variables  Included  the 
possibility  of  adverse  Impact  and  content  redundancy  and  the  lack  of  validity 
and  legal  defensIbIHty. 

Twenty-three  panels  of  12  subject-matter  experts  (l.e.i  one  panel  for  each 
civilian  job  family)  were  assembled  to  provide  a  content-  and  construct-valid 
promotion  system  that  would  consist  of  a  restricted  set  of  weighted  promotion 
factors.  Each  panel  performed  the  two  basic  tasks  of  (a)  selecting  6  to  10 
promotion  factors  frbm  the  list  of  24  candidate  factors  and  (b)  rank-ordering 
(sorting)  a  selected  random  sample  of  employees  in  terms  of  the  employees' 
promotabillty  as  reflected  by  these  factors.  Out  of  256  raters  *  only  two 
Inconsistent  raters  were  Identified;  data  from  these  raters  were  removed  from 
further  analyses.  Judgment  analysis  techniques  were  used  to  derive  a 
mathematical  algorithm  that  essentially  replicated  the  ranking  judgments  of 
pane)  members.  The  resulting,  promotion  factors  Identified  and  their  weights 
provided  unique  mathematical  combinations  (algorithms)  for  each  of  the  23  job 
families  by  supervisory  or  nonsupervlsory  category.  The  correlations  between 
the  panel  ranks*  and  the  ranks  assigned  by  the  weighted  algorithms  were  all 
above  .91*  with  most  above  .95.  Two  additional  panels  were  convened  to 
cross-valldat*  the  final  variable  selection,  ranking,  and  judgment  analysis 
process.  Cross-validation  correlations  ranged  from  .94  to  s98-.  Such  results 
provide  strong  support  that  CPAS  is  highly  content  and  construct  valid. 

The  Civilian  Potential  Appraisal  System  as  developed  by  AFHRL  was 
delivered  to  the  Office  of  Civilian  Personnel  Operations  In  April  1961'  and  was 
operationally  Implemented  In  April  1962. 


PREFACE 


It  Is  not  possible  to  recognize  Individually  the  Many  thousands  of 
civilian  and  Military  employees  who  have  devoted  so  Much  effort  to  developMent 
of  this  systeM  for  rating  and  ranking  civilian  eNployees.  However,  the 
project  officers  at  each  base  who  coordinated  the  C0NUS«w1de  collection  effort 
and  the  civilian  subject-Matter  specialists  who  participated  in  the  promotion 
data  panel  exercises  deserve  special  Mention  for  their  particular  dedication 
to  the  effort.  Their  dedication  Is  reflected  In  the  unusually  high  rate  of 
the  survey  returns  and  1r.  the  consistency  of  the  final  proeotlon  algorlthws. 

Several  Meobers  of  the  Office  of  Civilian  Personnel  Operations  at  Randolph 
Air  Force  Base  directly  Influenced  the  developMent  of  CPAS.  FroM  the  HuMan 
Resources  Laboratory,  Hr  Dick  Rogers  and  SSgt  Lee  Heyers  Must  also  be 
recognized  for  their  coeputer  expertise  and  tireless  efforts. 
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AIR  FORCE  CIVILIAN  POTENTIAL  APPRAISAL  SYSTEM: 

DEVELOPMENT 

I.  INTRODUCTION 

Background 

In  December  1976,  the  Directorate  of  Civilian  Personnel  (AF/MPK)  Initiated 
a  request  for  the  Air  Force  Hunan  Resources  Laboratory  (AFHRL)  to  develop  new 
appraisal  systems  for  the  civilian  work  force.  The  request  for  personnel 
research  (RPR),  originally  entitled  Supervisory  Appraisal  of  the  Current 
Performance  of  Air  Force  Civilian  Employees  (RPR  76-40),  was  validated  In 
July  1977,  and  the  development  process  began  Immediately.  The  request  was 
Initiated  because  the  existing  job  performance  and  promotion  appraisal  systems 
had  highly  inflated  ratings.  The  ratings  had  become  so  Inflated  that  It  was 
no  longer  possible  to  make  the  necessary  distinctions  among  employees  that 
were  required  for  promotion  and  other  personnel  actions.  Further,  the 
existing  systems  were  not  viewed  as  credible  by  the  employees,  supervisors,  or 
management. 

In  the  RPR,  AF/MPK  pointed  out  that  when  appraisal  Instruments  are 
‘developed  to  serve  a  variety  of  purposes  (e.g.,  selection  for  training, 
details,  reassignments,  merit  promotion),  their  use  for  the  latter  purpose  - 
merit  promotion  -  Inevitably  leads  to  their  Ineffectiveness  to  achieve  any  of 
these  purposes.”  The  use  of  appraisals  in  promotion  actions  places  such 
pressure  on  raters  to  Inflate  them  that  eventually  there  Is  Insufficient 
variance  In  the  ratings  to  allow  distinctions  to  be  made  between  or  among 
employees.  Innovative  modification  of  existing  methodologies  and  Instruments 
was  required,  but  the  modifications  had  to  conform  to  constraints  Imposed  by 
the  U.S.  Civil  Service  Commission  (now  the  Office  of  Personnel  Management)  and 
user  needs.  The  existing  Civil  Service  Commission  regulations  precluded: 

(1)  Withholding  Information  from  employees  on 
appraisal  ratings  or  any  other  Information  concerning 
their  job  performance;  (2)  Using  a  ‘single*  total 
score  for  each  employee  for  all  promotion 
considerations  (l.e.,  the  score  used  In  each 
consideration  must  be  composed  only  of  those  elements 
In  an  appraisal  form  that  are  relevant  to  the  position 
to  be  filled);  and  (3)  Using  elements  Inherently 
descriptive  of  the  ‘personality*  of  an  employee  rather 
than  as  attributes  relevant  to  job  performance  (RPR 
76-40,  p.  2). 

Additional  user-specified  (AF/MPK)  constraints  were  that  the  number  of 
forms  must  be  held  to  a  minimum,  required  forms  and  accompanying  Instructions 
were  to  be  as  simple  as  possible,  and  appraisals  were  to  be  compatible  for  use 
In  an  automated  data  processing  (ADP)  environment.  At  the  time  the  RPR  was 
received,  there  were  no  desired  Implementation  dates  specified  other  than  to 
replace  existing  deficient  systems  as  soon  as  practical. 
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The  Manpower  and  Personnel  Division  of  AFHRL  was  to  plan  and  conduct  the 
research  and  development  (R&D)  program.  Iteratively  develop  and  field- test 
prototype  system  components,  and  transition  the  systems  to  the  user.  The 
Directorate  of  Civilian  Personnel  and  Its  operational  arm,  the  Office  of 
Civilian  Personnel  Operations  (OCPO)  were  to  provide  policy  guidance  to  ensure 
that  the  developing  systems  were  operationally  Implementable;  provide  access 
to  test  subjects,  organizations,  and  existing  data  bases;  and  give  operational 
support  In  the  form  of  travel  funds  and  testing  facilities.  USAF/MPK  would  be 
fully  responsible  for  systems  Implementation,  Including  development  of 
regulations  and  operational  procedures. 

The  general  development  plan  was  presented  to  USAF/MPK  and  approved  In 
January  1976.  It  Included  development  of  separate  job  performance  evaluation 
and  promotion  appraisal  systems.  A  research-based  development  approach  would 
Include  basic  (6.1)-  and  exploratory  development  (6.2)  work  on  rating 
processes,  scale  development,  and  Identification  and  weighting  of  relevant 
experiences,  aptitudes,  and  performances.  The  R&D  results  would  then  be  used 
to  develop  the  performance  evaluation  measures  (job  performance  appraisal  and 
promotion  potential  appraisal)  that  would  be  used  operationally.  All  systems 
components  would  be  field-tested  and  revised,  based  on  objective  data. 
Analyses  would  be  designed  to  assess  employee  and  management  acceptability  and 
credibility  of  the  proposed  systems.  The  .systems  components  would  be 
validated  against  external  job  performance  measures,  and  an  operational  test 
and  evaluation  (OT&E)  would  be  performed  on  both  the  job  performance  and 
promotion  potential  appraisal  systems.  Each  OT&E  would  compare  existing  and 
proposed  systems  and  Include  a  cost-benefits  analysis  to  aid  management 
Implementation  decisions. 

In  November  1978,  the  passage  of  the  Civil  Service  Reform  Act  (CSRA-78; 
Public  Law  95-454)  changed  the  R&D  plans  previously  designed.  The  new  law 
required  development  of  three  separate  job  performance  appraisal  systems,  with 
specific  Implementation  dates  that  must  be  met.  To  meet  these  new 
requirements,  AFHRL  revised  their  ongoing  R&D  to  provide  the  three  systems  by 
the  specified  Implementation  dates  as  follows:  (a)  Senior  Executive  Appraisal 
System  (SEAS)  for  senior  executives:  September  1979  (Guerrlerl,  1981),  (b) 
General  Manager  Appraisal  System  (GMAS)  for  GM-13  through  GM-15s:  October  1980 
(Cowan,  Thompson,  &  Guerrlerl,  1983),  and  (c)  the  Job  Performance  Appraisal 
System  (JPAS)  for  all  non-manager,  non-SES  appropriated  fund  civilian 
employees:  October  1981  (Guerrlerl,  Thompson,  Cowan,  &  Gould,  1984).  It  1$ 
Important  to  note  that  CSRA-78  did  not  require  the  development  of  a  separate 
promotion  system  but  did  require  that  GMAS  and  JPAS  be  a  "basis*  for 
promotion.  To  Incorporate  the  use  of  GMAS  and  JPAS  as  a  basis  for  promotion, 
USAF/MPK  made  the  decision  that  only  employees  rated  "fully  successful"  or 
higher  In  their  current  jobs  would  be  considered  promotion  eligible.  Even 
though  CSRA  did  not  specifically  require  the  development  of  a  new  promotion 
system,  It  was  decided  that  a  new  promotion  system  was  still  needed.  The  new 
system  would  be  used  when  the  number  of  promotion  ellglbles  exceeded  the 
number  of  employees  that  could  be  readily  Interviewed  for  the  vacant 
position.  A  system  was  required  that  would  rate  employees  on  their  potential 
to  perform  In  higher-level  jobs,  and  these  ratings  could  then  be  used  to 
rank-order  the  promotion  ellglbles  for  management  review. 


Although  CSRA-78  Imposed  new  requirements  and  tight  Implementation 
suspenses,  USAF/MPK  requested  that  CPAS  development  continue  and  that  the 
final  product  be  delivered  at  the  same  time  as  JPAS.  With  these  Increased 
requirements  and  limited  R&D  resources,  many  proposed  R&D  projects  were 
cancelled  or  scaled  down,  and  development  relied  more  on  a  system  of 
field-testing  and  less  on  basic  research.  Nevertheless,  even  with  the 
Increased  workload,  CPAS  was  completed  and  delivered  to  OCPO  In  April  1981, 
and  implemented  in  April  1982. 

Overview 


This  paper  describes  the  development  of  the  requested  promotion  rating 
system,  the  Civilian  Potential  Appraisal  System  (CPAS).  The  body  of  the  paper 
describes  the  research  design,  the  process  used  to  organize  the  hundreds  of 
civilian  job  series  Into  job  families,  development  of  prototype  appraisal 
Instruments,  sample  selection,  the  data  collection  process,  the  reduction  of 
candidate  rating  factors  to  the  final  set,  policy-capturing  of  promotion  panel 
judgments,  development  and  cross-validation  of  promotion  algorithms,  and 
implementation  Issues.  Detailed  discussions  and  results  of  the  development 
and  analysis  processes  are  contained  In  the  appendices.  The  appendices 
present  the  same  general  Information  as  contained  In  the  main  body  of  this 
paper  but  go  Into  considerably  more  detail. 

Appendix  A  details  the  procedures  used  to  cluster  job  series  Into  job 
families,  lists  the  job  families,  and  Identifies  the  major  job  series  which 
make  up  each  job  family.  Appendix  B  describes  the  basic  research  conducted, 
the  development  of  prototype  appraisal  Instruments,  and  the  data  collection 
process,  along  with  copies  of  the  major  data  collection  Instruments.  Appendix 
C  describes  the  convening  of  promotion  data  panels,  procedures  used  In 
selecting  variables  and  ranking  promotion  eligibles,  the  process  which 
captured  the  panels'  ranking  policies,  and  the  development  of  the  operational 
CPAS  algorithms.  Appendix  D  provides  a  copy  of  the  operational  CPAS  rating 
form,  and  Appendix  E  contains  Instructions  to  supervisors  and  the  work  sheets 
to  be  used  during  the  rating  process. 

II.  APPROACH  AND  RESULTS 

The  major  objectives  of  the  proposed  research  design  were  (a)  to  review 
the  professional  performance  appraisal  literature  for  Identifying  promising 
components  and  scales  which  could  be  used,  and  to  consult  with  the  leading 
performance  measurement  experts  for  their  Input  to  the  developing  system,  (b) 
to  conduct  basic  research  to  evaluate  candidate  measures,  scaling  systems,  and 
novel  approaches,  (c)  to  develop  a  system  for  collapsing  the  large  number  of 
civilian  job  series  Into  a  manageable  number  of  homogeneous  job  families  for 
use  In  the  promotion  systems,  (d)  to  develop  a  wide  range  of  field-tested 
measurement  systems,  (e)  to  collect  data  from  a  large  Continental  United 
States  (CONUS)-wlde  sample  of  Air  Force  civilians,  (f)  to  convene 
job-famlly-speclflc  panels  of  experts  to  sit  as  promotion  panels  whose 
judgments  would  determine  the  final  promotion  factors  and  weighting  system, 
and  (g)  to  develop  and  cross-valldate  promotion  algorithms  by  which  CPAS  could 
replicate  the  experts'  promotablllty  judgments. 
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Review  of  Past  Research 


The  Implications  of  past  efforts  are  detailed  In  Appendix  B  and  cited 
publications.  In  sumary,  the  Initial  review  of  the  state-of-the-art  In 
performance  appraisal  (Mullins  &  Minn,  1979)  Identified  many  research  issues, 
cautions,  and  possible  approaches.  First,  no  ready-to-use,  satisfactory 
appraisal  system  for  Air  Force  use  existed.  In  developing  an  appropriate 
system,  discussions  with  expert  Industrial  psychologists  (Mullins,  1981; 
Mullins  A  Minn,  1979)  confirmed  that  ratings  used  for  promotion  purposes 
should  be  as  objective  and  job  related  as  possible.  Asking  supervisors  to 
rate  directly  employees'  potential  to  perform  higher  level  jobs  with  which  the 
raters  themselves  were  not  familiar  would  not  be  technically  sound  or  legally 
defensible.  Further,  there  was  little  empirical  evidence  that  supervisors  are 
able  to  make  satisfactory  projections  of  future  performance.  On  these 
grounds.  It  appeared  that  job-relevant  aptitude  scores  and  verifiable 
experience,  training,  and  other  demographic  Information  would  probably  be  the 
most  promising  objective  predictors  of  future  performance.  A  combination  of 
peer,  supervisor,  and  alternate  supervisor  ratings  of  current  job  performance 
might  also  be  used  In  the  prediction  system.  Mhereas  the  user  asked  for  a 
system  that  would  rate  employees'  potential.  It  soon  became  apparent  that  only 
measures  of  past  experience  and  current  performance  would  be  appropriate.  The 
new  promotion  system  thus  had  to  identify  and  use  current  and  past 
performances  that  would  be  predictive  of  performance  at  higher  levels,  rather 
than  directly  measuring  future  performance  potential. 

Military  selection  and  promotion  systems  have  been  exempted  from  the 
jurisdiction  of  civil  law  by  the  Office  of  Management  and  Budget,  but  similar 
systems  for  Department  of  Defense  civilians  have  no  such  exemptions.  The  mood 
of  the  nation  for  equal  employment  opportunity  and  the  growing  number  of  legal 
challenges  of  civilian  promotion  systems  Indicated  a  growing  probability  that 
any  promotion  system  would  eventually  be  challenged  In  court  (United  States 
Office  of  Personnel  Management,  1979).  Therefore,  the  CPAS  developers  made  an 
Initial  assumption  that  the  system  they  developed  would  be  challenged  In 
court.  A  top  priority,  then,  was  to  develop  a  legally  defensible  and  valid 
system. 

The  Uniform  Guidelines  for  Employee  Selection  of  1978  became  available  In 
1979,  and  early  court  cases  began  operationally  defining  the  meanings  of  the 
provisions  (Casclo  &  Bernardln,  1981).  In  effect,  the  Guidelines  say  that  a 
selection  or  promotion  system  which  creates  adverse  impact  against  a  minority 
or  protected  group  of  employees  or  potential  employees  Is  legally  defensible 
only  If  the  systems'  development  process  considered  nil  practical  variables 
that  are  likely  to  be  related  to  job  performance  and  selected  those  which  are 
most  valid  (content,  construct,  or  predictive  validity)  but  are  least  likely 
to  create  adverse  Impact.  Hence,  the  design  concept  for  developing  CPAS  was 
to  obtain  the  maximum  Information  on  each  employee  from  a  variety  of  different 
sources  and  Identify  the  most  efficient  subset  that  would  minimize  adverse 
Impact  while  providing  a  valid  system.  Since  originally  there  were  no 
required  Implementation  dates  and  because  so  many  promotion  rating  issues 
remained  to  be  answered,  the  design  concept  was  essentially  a  research-based 
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system  where  the  empirical  data  rather  than  subjective  judgments  of  the 
developers  would  determine  the  final  form  of  the  Civilian  Potential  Appraisal 
System.  To  meet  this  objective,  a  system  of  three  diverse  appraisal  sources 
was  developed  --  testing,  background  Information,  and  ratings. 

Basic  Research  Conducted 


The  mass  of  research  on  job  performance  ratings  has  failed  to  demonstrate 
that  any  specific  set  or  type  of  rating  statement  significantly  enhances 
prediction  accuracy.  For  example,  behavlorally  anchored  rating  scales  (BARS) 
have  been  extensively  researched  In  recent  years.  However,  evidence  for  the 
psychometric  superiority  of  these  scales  Is  mixed,  and  their  predictive  power 
Is  virtually  untested  (Ounnette  &  Borman,  1979).  In  support  of  the  RPR,  a 
series  of  AFHRL  studies  (Massey,  Mullins,  &  Earles,  1976)  also  failed  to 
detect  any  difference  between  worker-,  task-,  and  trait-oriented  rating  scales 
In  predicting  Noncommissioned  Officer  (NCO)  Academy  performance.  Another 
series  of  studies  (Curton,  Ratliff,  S  Mullins,  1979)  likewise  found  no 
differences  between  sets  of  rating  factors,  even  though  subject-matter  experts 
had  preselected  separate  sets  of  factors  that  seemed  most  and  least  relevant 
to  performance.  Although  alternative  rating  contents  or  techniques  may 
ultimately  provide  superior  ratings,  these  scales  have  not  been  developed. 

At  this  point  (early  1979),  an  ongoing  series  of  studies  Investigating  the 
use  of  peer  ratings  and  novel  scales  which  were  not  subject  to  classical  halo 
and  rating  Inflation  was  discontinued.  The  requirement  to  develop  three 
separate  job  performance  appraisal  systems  with  tight  suspenses  (CSRA-78) 
forced  a  reduction  of  scientists*  efforts  on  basic  research.  In  addition, 
funding  for  this  stream  of  basic  (6.1)  research  was  not  renewed.  It  Is  not, 
however,  likely  that  the  resulting  CPAS  system  would  have  differed 
significantly  from  Its  present  form  had  the  research  continued  because  of  the 
time  required  to  conduct  and  transition  basic  research  and  the  due  date  for 
CPAS.  Research  completed  by  1979  did,  however,  provide  the  rating  scales  used 
In  the  final  development  of  CPAS,  as  well  as  provide  many  components  of  the 
research  design  used  In  developing  the  operational  CPAS. 

Development  of  Job  Family  Clusters 


There  are  approximately  1,500  civilian  job  series  In  the  Air  Force.  It 
was  necessary  to  cluster  these  job  series  Into  a  small  number  of  groups  or  job 
families  before  development  of  CPAS  could  proceed.  If  CPAS  was  to  use 
relevant  past  experience,  aptitude,  and  current  job  performance  measures  to 
predict  future  performance  In  higher-level  jobs,  candidate  jobs  had  to  be 
grouped  Into  homogeneous  clusters  based  on  job  requirements.  Once  jobs  were 
so  clustered,  relevant  requirements  could  be  determined  and  promotion 
candidates  rank-ordered  according  to  the  degree  to  which  they  met  those 
requirements. 

To  obtain  a  manageable  number  of  logical  and  homogeneous  clusters  of  job 
types,  six  position  classifiers  and  research  psychologists  were  divided  Into 
two  Independent  panels.  They  were  Instructed  to  arrange  the  1,500  Air  Force 
civilian  job  series  Into  some  20  to  30  homogeneous  clusters  based  on  task 


subject  latter  and  job  requirements.  The  clusters  developed  by  the  two  panels 
were  then  compared  and  differences  resolved.  The  comparisons  Indicated  that 
there  was  near-perfect  overlap  between  the  clusters  developed  by  the  two 
panels.  The  only  difference  was  In  the  number  of  trades-and-crafts  job 
families,  which  one  panel  had  separated  based  on  environment  and  the  other 
panel  had  combined  on  the  grounds  that  their  job  tasks  were  similar.  After 
resolution  of  these  differences  In  a  joint  meeting,  23  job  family  groupings 
were  specified.  There  were  three  professional,  seven  technical,  five 
administrative;  two  .  clerical,  and  six  trades-and-crafts  job  families. 
Appendix  A  lists  the  23  job  families  and  the  major  occupational  series  making 
up  each  job  family.  Also  Included  Is  a  more  detailed  description  of  the  panel 
process  and  a  summary  report. 

Data  Collection  Instruments 

Overview.  No  measurement  Instruments  currently  available  were  considered 
adequate  for  use  In  the  CPAS.  Therefore,  three  experimental  data  collection 
Instruments  were  developed:  the  Demographic  Questionnaire-Worker,  the 
Job/Worker  Characteristics  Ratings  booklet,  and  the  Civilian  Personnel 
Examination  and  Score-Checking  Test. 

Literature  surveys,  review,  and  evaluation  of  currently  existing 
Government  and  Industry  rating  forms,  job  analysis  data  from  similar  military 
occupations,  and  subjective  judgments  by  the  research  team  provided  a  pool  of 
potential  Items  and  scales  from  which  the  first  two  measurement  Instruments 
were  developed.  Test  Instruments  previously  developed  for  Air  Force  enlisted 
personnel  provided  the  basic  prototypes  for  the  aptitude  measures. 

The  Demographic  Questionnaire-Worker  was  a  102-Item,  self-report  survey 
designed  to  collect  traditional  demographic  information  on  each  employee,  such 
as  age,  sex,  and  ethnic  background;  promotion,  education,  and  job  training 
history;  current  job  data  such  as  time  In  service,  grade  level,  and  on-the-job 
performance  awards;  and  Interest  and  motivation  toward  acceptance  of 
additional  responsibility,  training  and  effort  to  become  eligible  for 
promotion. 

The  Job/Worker  Characteristics  Ratings  Booklet  contained  108  Items,  with 
9-polnt,  adjectival,  anchored  rating  scales  designed  to  obtain  ratings  on  a 
wide  range  of  operationally  defined  worker/job  performance  skills  and 
knowledges.  Also  Included  In  this  booklet  were  ratings  of  overall  job 
performance,  predicted  performance  In  the  next  higher  job,  and  estimates  of 
the  scores  the  worker  was  likely  to  obtain  on  paper-and-pencll  measures  of 
verbal,  quantitative,  mechanical,  electronics,  and  spatial  aptitude. 

The  Civilian  Personnel  Examination  (CPE)  and  the  Score-Checking  Test  (SCT) 
were  designed  to  measure  objectively  the  workers'  quantitative,  verbal, 
spatial,  mechanical,  electronics,  and  administrative  aptitudes.  Although 
these  two  Instruments  were  Initially  developed  to  be  part  of  the  operational 
CPAS,  the  aptitude  tests  were  not  retained  as  candidate  components  for  the 
operational  promotion  system.  However,  they  were  used  In  the  experimental 


development  process  to  provide  a  Beans  for  assessing  rater  accuracy  and 
determining  the  reliability  of  specific  skills  and  knowledge  ratings  given  to 
employees. 

Demographic  Questionnaire-Worker.  An  employee  self-report  demographic 
questionnaire  was  believed  to  be  the  most  efficient  Beans  of  obtaining  certain 
very  basic  data  Important  for  a  promotion  system.  A  long  history  of  studies 
on  demographic  and  other  self-report  type  questionnaires  has  demonstrated 
their  validity  and  usefulness  In  predicting  future  performance  (Reilly  &  Chao, 
1980).  Reilly  and  Chao  reported  that  self-report  background  measures  provide 
as  much  or  more  validity,  with  as  little  or  less  adverse  impact,  than  do  many 
alternative  selection  devices.  Therefore,  these  demographic  data  seemed  to  be 
a  valuable  source  for  appraisal  Information.  Many  of  the  Items  Included  In 
the  questionnaire,  such  as  race  and  sex,  had  no  potential  for  actual  use  In 
CPAS  but  were  Included  to  aid  analysis  of  the  obtained  data  for  potential 
adverse  Impact  against  minorities  or  to  answer  research  questions  In  later 
analyses.  The  complete  Demographic  Questionnaire-Worker  Is  shown  In  Appendix 
B-2. 


Job/Worker  Characteristics  Ratings.  The  primary  purpose  of  the  Job/Worker 
Characteristics  Rating  booklet  was  to  obtain  job  performance  ratings  and  rater 
accuracy  data.  Because  this  booklet  played  the  key  role  In  development  of  the 
final  CPAS,  the  design  of  the  booklet  Is  covered  In  considerable  detail  In 
Appendix  B.  A  full  copy  of  the  booklet  Is  Included  In  Appendix  B-3. 

As  previously  mentioned,  past  research  with  rating  factor  content  and 
scales  has  Indicated  that  as  long  as  the  components  are  constructed  with  care, 
ihe  specific  format  seems  to  make  little  difference.  One  possible  exception 
Is  the  number  of  scale  points.  When  rating  scale  points  are  well  defined, 
reliability  usually  Increases  as  the  number  of  points  Increase,  but  with 
diminishing  returns  after  nine  points  (Gould,  1978).  In  view  of  these 
findings,  neither  the  format  nor  the  job-specific  content  of  the  rating  scales 
was  heavily  emphasized.  Instead,  a  working  group  of  AFHRL  behavioral 
scientists  met  In  November  1978,  to  select  a  set  of  general  performance 
ratings  and  a  set  of  appraisal  factors  that  would  apply  to  all  civilian  job 
families  in  the  Air  Force.  It  was  not  Intended  that  every  factor  would  apply 
to  every  job  family.  These  ratings  and  factors  are  similar  to,  and  extracts 
of,  those  currently  used  In  a  number  of  research  and  operational  systems 
(Massey,  Mullins,  &  Earles,  1978;  Roach,  1956).  The  ratings  booklet  contains 
a  variety  of  Items  In  10  sections  and  was  designed  to  be  completed  by  raters 
who  were  acquainted  with  the  employee's  job  performance.  Each  of  the  10 
sections  tapped  a  different  type  of  Information;  e.g.,  overall  job 
performance,  ratings  on  trait  factors,  ratings  on  general  abilities,  and  ratee 
background  Information.  Except  for  some  Ipsatlve  scales,  experimental  scales, 
and  the  ratee  background  sections,  ratings  were  made  on  9-polnt,  Likert-type 
normative  scales,  with  anchors  given  at  each  point.  Ipsatlve  scales  use  the 
person  being  rated  as  the  reference  point  and  rank-order  the  rating  Items  (In 
this  case,  appraisal  factors)  for  that  person  from  high  to  low  to  reflect 
their  strengths  and  weaknesses.  Normative  scales  solicit  ratings  of  how  a 
person  compares  to  a  reference  group  (usually  a  peer  group),  and  the  ratings 
are  in  terms  of  relative  standing. 
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General  performance  ratings  are  presented  In  Section  I  of  the 
Characteristics  Ratings  booklet.  Each  booklet  section  Mas  designed  for  a 
specific  purpose.  Sections  I  and  II  provide  normative  job  performance 
ratings.  They  provide  for  the  ratee's  evaluation  of  the  worker's  current  job 
performance,  potential  for  performance  at  the  next  grade  level,  and 
supervisory  and  managerial  capability.  Section  III  contains  12  traits  that 
were  chosen,  from  hundreds  available,  according  to  their  judged  applicability 
to  most  Air  Force  civilian  jobs,  comprehensiveness,  and  understandablllty. 
Ratings  were  limited  to  12  since  this  was  considered  the  greatest  number  of 
traits  raters  could  distinguish  between  (Mullins,  Meeks,  S  Mllbourn,  1978). 

Numerous  studies  have  demonstrated  the  generallzablllty  of  rater  accuracy 
across  different  rating  dimensions  (Cummings,  1980).  Estimates  of  rater 
accuracy  are  needed  for  the  legal  defense  of  ratings;  therefore.  Sections  II 
and  IV  were  designed  to  provide  rater  accuracy  Indices.  Section  II  contained 
a  single  Item  requiring  the  rater  to  estimate  the  worker's  score  on  the  Word 
Knowledge  portion  of  the  CPE.  Section  IV  consisted  of  ratings  on  eight 
general  abilities  reflecting  seven  of  the  11  CPE  subtests.  These  ability 
ratings  provide  rater  accuracy  Indices  when  compared  with  the  ratee's  CPE  and 
SCT  scores. 

The  recurring  Inflation  of  normative  performance  ratings  In  operational 
settings  suggested  that  new  evaluation  techniques  needed  to  be  Investigated. 
Cne  such  technique  Is  the  Ipsatlve  ranking  technique.  With  Ipsatlve  scales, 
the  rater  ranks  the  traits  In  a  given  list  according  to  which  traits  are  the 
worker's  best.  A  similar  ranking  Is  obtained  for  the  worker's  job  position 
(l.e.,  which  traits  are  the  most  Important  for  successful  job  performance). 
The  two  sets  of  rankings  are  then  combined  to  yield  a  job-worker-match 
coefficient.  This  coefficient  can  be  used  In  combination  with  other  rating 
and  background  data  as  a  possible  Index  of  promotion  potential.  Therefore,  to 
reduce  rater  Inflation,  Sections  V,  VI,  VIII,  and  IX  were  Included  to  provide 
the  Ipsatlve  scales  needed  to  compute  two  different  Inflation-free, 
job-worker-match  Indices.  During  analysis,  the  Information  from  these  four 
sections  was  to  be  combined  to  compute  a  job-worker  trait  match  and  a 
Job-worker  ability  match.  The  Ipsatlve  rankings  could  also  be  combined  with 
the  normative  ratings  to  yield  a  variety  of  Ipsatlve-normatlve  measures.  For 
example,  the  six  traits  most  Important  for  successful  job  performance  could  be 
Identified  by  the  rankings  and  only  these  six  ratings  used  as  predictors. 
Mullins  and  Meeks  (1979)  provide  a  technique  for  the  "normative  use  of 
Ipsatlve  ratings." 

To  assess  an  Individual's  motivation  and  drive  toward  promotion,  Section 
VII  Included  rater  estimates  of  these  attributes.  The  ratings  were  not  firmly 
grounded  In  previous  research  but  were  hypothesized  to  provide  valuable 
predictive  Information.  Items  1  through  4  provided  for  Information  on  the 
worker's  on-the-job  drive  and  productivity  level:  speed  of  work, 
productivity,  percent  time  spent  working,  and  Initiative.  Each  rating  was 
made  on  a  5-polnt,  Likert-type  scale,  with  anchors  given  at  each  point.  Items 
5  through  21  provided  for  the  rater's  assessment  of  the  worker's  desire  for  a 
promotion.  Each  Item  required  the  rater  to  Indicate  (yes/no)  whether  the 
worker  would  be  willing  to  Incur  a  certain  cost  In  order  to  be  promoted  to 
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the  next  grade  level.  Such  costs  included:  entering  a  training/retraining 
program,  moving  to  another  organization,  accepting  a  job  outside  the  worker's 
current  job  series,  assuming  additional  duties,  and  performing  additional 
supervisory  tasks. 

The  Section  VII  ratings  served  two  Important  functions.  First,  both  types 
of  measures  were  later  tested  as  potential  predictors  of  a  ratee's  suitability 
for  promotion.  Second,  the  'desire  for  promotion"  measures  (Items  5  through 
21)  provided  an  additional  Index  of  rater  accuracy.  In  the  Demographic 
questionnaire,  the  workers  Indicated  whether  they  desired  a  promotion  on  these 
same  17  Items.  By  comparing  the  rater's  responses  on  these  17  items  with  each 
worker's  responses  on  the  Demographic  Questionnaire,  It  was  possible  to 
determine  how  well  each  rater  knew  the  worker's  desire  for  a  promotion. 

Section  X  contained  20  background  Items  that  related  to  the  rater's 
familiarity  with  the  Individual.  These  Items  were  designed  to  serve  as 
possible  control  measures  for  prediction  purposes  and  to  be  studied  as 
correlates  for  the  Investigations  of  rater  accuracy  and  possible  adverse 
impact. 

Civilian  Personnel  Examination  and  Scoring-Checking  Test.  Aptitude  tests  were 
considered  an  important  candidate  component  for  the  proposed  operational  CPAS 
system.  Such  aptitude  tests  have  been  widely  used  and  found  to  be  valid 
predictors  of  training  outcomes  and  job  performance  In  a  wide  variety  of 
situations  (Green,  1978).  Aptitude  tests  can  cover  a  broad  spectrum  of 
abilities,  are  not  as  subject  to  Inflation  as  are  ratings  (although  they  may 
be  subject  to  bias),  and  are  generally  more  objective  measures  of  an 
Individual's  abilities  than  supervisors'  subjective  assessments  and 
evaluations.  In  addition,  the  use  of  aptitude  tests  may  provide  an  Indication 
of  skills  not  necessarily  required  on  an  Individual's  current  job,  hence  not 
observable  or  ratable  by  supervisors,  but  which  are  essential  In  a 
higher-level  job  such  as  supervisory  and/or  managerial;  for  example,  a  Federal 
Wage  System  flightline  maintenance  employee  probably  uses  few  verbal  and 
administrative  skills  In  his  job,  his  supervisor  would  be  marginally  qualified 
to  rate  such  skills,  and  yet  those  skills  would  be  essential  If  the  employee 
was  promoted  to  the  next  level  (l.e..  Wage  Supervisor),  For  these  reasons, 
aptitude  tests  were  Included  In  the  original  experimental  design  for 
developing  CPAS.  As  will  be  explained  later,  aptitude  tests  were,  however, 
not  Included  In  the  operational  system. 

Two  aptitude  tests,  the  Civilian  Personnel  Examination  (CPE)  and  the 
Score-Checking  Test  (SCT)  were  developed.  The  CPE  was  a  modification  of  the 
Airman  qualification  Examination,  Form  J  (AQE-J)  (Vltola,  Massey,  &  Wllbourn, 
1971),  which  was  updated  and  adapted  for  civilian  personnel.  The  examination 
consisted  of  11  subtests,  measuring  verbal,  quantitative,  mechanical,  and 
electronics  ability.  The  subtests  most  heavily  emphasized  blue-collar  and 
clerical  skills,  although  many  of  the  subtests  (e.g.,  reading  ability, 
arithmetic  reasoning,  word  knowledge)  could  apply  to  a  variety  of  job  series. 
Appendix  B  (particularly  Table  B-2)  details  the  CPE  and  Its  development.  The 
Score-Checking  Test  (SCT),  a  carefulness  test  (Mullins  A  Force,  1962)  which 
required  the  examinee  to  check  two  sets  of  numbers  for  transcribing  errors. 


was  designed  to  evaluate  the  examinee's  carefulness  in  spotting 
discrepancies.  Carefulness  is  an  essential  entry-level  ability  for  most 
clerical  and  administrative  jobs  but  is  not  measured  by  most  standard  aptitude 
tests. 

Instruments  Pretest.  Initial  pretests  of  early  versions  of  the 
Demographic  Questionnaire-Worker  and  the  Job/Worker  Characteristics  Ratings 
booklet  were  conducted  with  small  samples  of  employees  at  Lackland  AFB  (N=15) 
and  Randolph  AFB  (N"20).  Pretesting  provided  the  opportunity  to  evaluate  the 
readability  of  the  materials,  the  ease  of  following  Instructions,  and  any 
difficulties  experienced  In  using  the  scales.  Feedback  received  led  to 
numerous  wording  revisions,  changes  In  some  rating  scales,  addition/deletion 
of  some  Items,  and  reordering  of  sections. 

Full-scale  pretests  of  the  CPAS  data  collection  materials  were  conducted 
at  McClellan  AFB  (February  -  March  1979)  and  Patrick  AFB  (May,  1979)  using  all 
three  parts  of  the  full  system  (testing,  demographics,  and  ratings).  Complete 
data  sets  were  collected  for  266  workers  at  McClellan  AFB  and  for  102  workers 
at  Patrick  AFB.  The  means  and  standard  deviations  of  the  ratings  Indicated 
that  there  were  no  serious  Inflation  or  restriction  In  range  problems. 
Workers  completed  the  CPE,  SCT,  and  Demographic  Questionnaire-Worker  In  a 
single,  half-day  testing  session.  Approximately  1  month  after  the  testing 
session,  the  rating  booklets  were  mailed  out  to  the  prospective  raters.  In 
these  pretests,  three  ratings  were  used  for  each  employee:  the  supervisor,  a- 
co-worker,  and  an  associate  of  the  supervisor. 

Follow-up  Interviews  at  these  bases  revealed  that  a  few  people  found  the 
rank-ordering  procedure  confusing;  others  found  It  difficult  to  rank-order  the 
12  traits  (Sections  V  and  VIII)  accurately.  A  few  raters  complained  that  the 
booklet  took  too  long  to  complete,  though  most  raters  agreed  that  It  took 
approximately  1  hour.  In  general,  most  raters  found  the  rating  booklet 
straightforward  and  easy  to  work  with. 

The  follow-up  Interviews  also  Indicated  that  supervisors  and  employees 
tended  to  feel  tl:e  evolving  CPAS  system  would  be  superior  to  the  current 
system.  The  majority  of  participants  believed  that  a  system  of  objective 
tests,  comprehensive  ratings,  and  multiple  raters  provided  the  basis  for  a 
better  system.  Several  participants  particularly  favored  having  multiple 
raters  so  that  someone  In  addition  to  their  supervisor  would  be  Involved  In 
evaluating  their  performance. 

As  a  result  of  the  field  test,  minor  modifications  were  made  In  all  three 
components  of  the  CPAS  In  preparation  for  the  CONUS-wlde  data  collection 
effort.  A  list  of  formal  training  sources  (Items  68  to  102)  was  added  to  the 
Demographic  Questionnaire-Worker,  and  the  order  of  most  of  the  Items  In  the 
questionnaire  was  changed  to  present  a  more  logical  sequence  to  the  examinee. 
With  these  modifications,  the  CPAS  materials  were  forwarded  for  review  and 
approval  by  the  Air  Force  Directorate  of  Civilian  Personnel  (USAF/MPK)  prior 
to  their  use  In  the  major  CONUS-wlde  data  collection  process. 
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Management  Review  of  the  Proposed  System.  In  October  1979,  the  prototype 
Civilian  Potential  Appraisal  System  to  be  used  In  the  CONUS-wlde  field  test 
and  data  collection  effort  was  reviewed  by  the  Air  Force  Directorate  of 
Civilian  Personnel  (USAF/HPK).  The  experimental  prototype  system  proposed  by 
the  developers  consisted  of  the  CPE,  SCT,  Demographic  Questionnaire-Worker, 
and  the  Job/Worker  Characteristics  Ratings  booklet,  which  was  to  be  completed 
by  three  raters  on  each  employee  (supervisor,  co-worker,  and  supervisor's 
alternate). 

Seven  data  collection  alternatives  were  actually  presented  to  the  user 
with  benefits  and  costs  In  man-hours,  as  well  as  an  assessment  In  terms  of  the 
research  Impact.  The  alternatives  ranged  from  no  peer  or  supervisor's 
alternate  ratings  and  no  aptitude  tests  to  three  sets  of  ratings  and  aptitude 
tests  for  all  target  employees.  USAF/MPK  selected  a  combination  of  20,000 
target  workers  with  supervisor  and  co-worker  ratings.  Some  2,000  of  the 
workers  would  be  given  the  CPE  and  SCT. 

The  primary  concerns  expressed  by  USAF/MPK  focused  on  the  legal 
acceptability  of  the  various  components  of  the  proposed  system  and  the  number 
of  man-hours  Involved  in  collecting  the  data.  As  a  result  of  their  decision, 
the  CPE,  SCT,  and  ratings  by  the  co-worker  and  the  supervisor's  alternate  were 
eliminated  from  consideration  In  the  operational  system.  USAF/MPK  eliminated 
the  tests  because  of  the  controversy  surrounding  the  use  of  paper-and-pencll 
tests  for  selectlon/promotlon  purposes  at  that  time  (United  States  Office  of 
Personnel  Management,  1979)  and  because  the  Carter  administration  had  just 
signed  a  consent  decree  to  eliminate  the  PACE  examination  (an  aptitude  test 
similar  to  the  CPE)  for  Civil  Service  entrants  because  of  documented 
differences  In  performance  between  minority  and  majority  applicants.  With  the 
Government's  decision  not  to  defend  a  demonstrably  valid  test.  It  was  felt  the 
Air  Force  should  not  undertake  the  use  of  a  similar  test  In  an  appraisal 
system. 

A  second  factor  In  the  decision  to  eliminate  the  tests  from  CPAS  was  the 
number  of  man-hours  Involved  In  testing.  Full  testing  would  require 
approximately  4  hours  (or  more)  per  employee.  This  requirement  was  considered 
prohibitive  even  for  the  CONUS-wlde  data  collection  effort. 

The  co-worker  ratings  and  supervisor's  alternate  ratings  were  eliminated 
from  operational  CPAS  consideration  because  of  the  questionable  use  of  these 
Individuals  as  official  raters  for  the  employee.  It  was  believed  that 
operational  use  of  ratings  by  Individuals  who  are  not  official  raters  could 
create  serious  legal,  administrative,  and  morale  problems  for  the  promotion 
system.  Therefore,  rating  sources  other  than  direct  first-line  supervisors 
were  eliminated  from  Inclusion  In  the  final  operational  system. 

Although  the  aptitude  tests  and  nonsupervlsory  ratings  were  deleted  from 
consideration  in  the  operational  system,  It  was  decided  that  up  to  10  percent 
of  the  CONUS  sample  would  be  given  the  CPE  and  SCT  In  addition  to  the  other 
materials  for  research  purposes.  These  data  would  permit  an  examination  of 
rater  accuracy,  test  validity,  and  evaluation  of  possible  adverse  Impact.  In 
addition,  the  co-worker  ratings  were  also  to  be  collected  for  all  ratees  for 


research  purposes  only.  This  arrangement  provided  a  comparison  of  co-worker 
versus  supervisor  rater  accuracy,  as  well  as  an  additional  rating  source  for 
possible  validity  studies.  The  supervisor's  alternate  ratings,  however,  were 
eliminated  entirely  from  the  research  design. 

CONUS-Wlde  Data  Collection 


A  CONUS-wlde  field  test  was  originally  scheduled  to  be  accomplished  from 
November  1979  to  January  1960.  However,  problems  encountered  with  the 
American  Federation  of  Government  Employee  (AFGE)  representatives  within  Air 
Force  Logistics  Command  delayed  completion  of  the  project  until  June  1980. 1 
The  purpose  of  the  CONUS-wlde  effort  was  to  obtain  sufficient  data  on  all 
candidate  CPAS  components  to  permit  reduction  of  the  candidate  variables  to  a 
manageable  subset  and  to  Identify  relevant  variables  and  determine  their 
relative  weighting  for  rank-ordering  promotion  ellglbles  within  each  job 
family. 

A  stratified  random  sample  of  20,000  target  workers  from  the  74  most 
populous  CONUS  bases  was  selected.  The  sample  was  stratified  by  grade,  job 
series,  sex,  and  ethnic  category  with  600  to  1,100  cases  representing  each  of 
the  23  job  families.  The  sample  over-represented  the  number  of  minority 
members  so  that  cell  sizes  would  be  adequate  for  bias  and  adverse  Impact 
analyses. 

Demographic  Questionnaire-Worker  booklets  were  sent  to  20,000  target 
workers  along  with  a  self-addressed  envelope  to  return  the  survey  directly  to 
AFHRL .  Such  a  procedure  would  ensure  respondents  that  privacy  of  the  data 
requested  from  each  Individual  would  be  maintained.  Supervisors  and 
co-workers  of  each  of  the  target  employees  also  received  a  Job/Worker 
Characteristics  Ratings  booklet  to  be  completed  and  returned  In  a 
self-addressed  envelope  to  AFHRL. 


Although  national  representatives  of  the  AFGE  had  been  Informed  on  two 
occasions  about  CPAS  development,  AFGE  representatives  within  AFLC  continued 
to  be  strongly  opposed  to  the  development  of  a  system  based  on  ranking  factors 
other  than  longevity.  After  three  bargaining  sessions  failed  to  allay  their 
concerns,  USAF/NPK  directed  that  the  data  collection  effort  proceed  within 
AFLC  as  It  had  within  the  other  commands. 
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The  CPE  was  administered  to  a  random  sample  of  2,000  of  the  20,000  target 
workers  to  obtain  aptitude  measures  for  the  rater  accuracy  studies. 
Seventy-one  percent  of  the  mall-out  materials  were  returned,  with  65%  of  the 
target  workers  (N=12,865)  having  completely  matched  and  usable  data  sets. 
Possibly  due  to  the  AFGE  concerns  over  the  development  of  a  new  promotion 
system  which  considered  any  factors  other  than  seniority  to  achieve  promotion 
eligibility  status,  the  return  rates  for  the  blue-collar  civilian  workers  were 
significantly  below  those  of  other  Federal  service  employees.  However,  the 
overall  distribution  of  the  blue-collar  returns  was  still  adequate  for  full 
analysis  of  the  data.  Substantial  variance  In  the  behavioral  performance 
element  ratings,  constructive  write-in  comments,  and  completeness  of  the  data 
attest  to  the  overall  support  that  workers  and  supervisors  gave  to  the 
promotion  system  development  effort.  Statements  Indicating  the  use  of  the 
ratings  for  "research  purposes  only"  and  an  explanation  of  the  nature  and 
purpose  of  the  entire  project  and  the  dedicated  efforts  of  local  base  project 
monitors  were  responsible  for  the  enthusiastic  support  of  most  participants. 
A  more  detailed  description  of  the  sample  selection  process,  sample 
characteristics,  administration  procedures,  data  collection  and  processing, 
and  analysis  of  returns  Is  included  In  Appendix  B. 

Variable  Reduction 


The  210  data  elements  In  the  Demographic  Questionnaire-Worker  (102  Items) 
and  Job/Worker  Characteristics  Ratings  booklets  (108  Items)  presented  a 
formidable  array  of  candidate  variables  for  use  In  the  operational  CPAS. 
Although  Promotion  Policy  Data  Panels  of  experienced  personnel  were  to  be  used 
to  Identify  variables  specifically  related  to  performance  In  their  job  family, 
they  could  not  be  expected  to  handle  such  a  large  pool  of  Items.  Since  these 
panel  members  could  not  be  expected  to  evaluate  the  Items  on  their 
psychometric  characteristics  and  legal  Implications,  a  panel  composed  of 
industrial  psychologists  (under  contract  and  from  AFHRL )  and  personnel 
specialists  (from  OCPO)  was  tasked  to  reduce  the  pool  of  variables.  These 
panel  members  were  knowledgeable  of  the  personnel  system,  applicable  case  law, 
and/or  psychometrics. 

Since  a  high  proportion  of  selection  systems  have  been  determined  to 
create  adverse  Impact  on  minority  groups  (Casclo  &  Bernardln,  1981,  United 
States  Office  of  Personnel  Management,  1979)  particular  attention  was  paid  to 
the  requirements  of  the  Uniform  Guidelines  for  Employee  Selection  during  the 
variable  reduction  phase.  The  guidelines  say.  In  effect,  that  a  selection 
system  which  creates  adverse  Impact  Is  defensible  If  the  development  process 
tried  to  Identify  all  sources  of  Information  that  could  be  related  to  job 
performance  and  then  selected  for  operational  use  those  Information  Items 
which  minimized  adverse  Impact;  but  only  If  the  selected  set  of  Items  provides 
a  highly  valid  selection  system.  In  other  words,  the  goal  Is  to  try  to  select 
Items  which  minimize  adverse  Impact  but  do  not  sacrifice  system  validity  In 
the  process. 

To  reduce  the  210  demographic  and  rating  variables  to  a  workable  subset,  a 
large  data  matrix  was  constructed  for  rating  each  of  the  elements  on  a  series 
of  selection  criteria.  Four  basic  types  of  criteria  were  used:  (a)  estimated 
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validity,  (b)  uniqueness,  (c)  bias,  and  (d)  legal  defenslblllty.  Correlations 
of  the  variables  with  peer  ratings  of  overall  performance  and  projected 
performance  at  the  next  higher  levels  served  as  estimates  of  Item  validity.  A 
computing  process  titled  VARSEL  (Qould  &  Chrlstal,  1976),  was  used  to  develop 
Indices  of  Item  uniqueness  and  redundancy  from  an  Item  Intercorrelation 
Matrix.  Comparisons  of  mean  ratings  by  race  and  sex  category  were  used  to 
evaluate  Item  bias.  Thus,  objective  measures  of  each  Item’s  validity, 
uniqueness,  and  bias  were  available.  Subjective  ratings  of  legal 
defenslblllty  were  made  by  personnel  specialists  and  psychologists,  based  on 
case  law. 

After  the  objective  measures  were  available,  a  panel  of  eight  research 
psychologists  and  two  personnel  specialists  was  convened  to  select  a  workable 
subset  of  variables  from  the  data  matrix  using  the  criterion  indices  for 
guidance.  All  data  Items  were  rated  on  a  continuum  of  their  legal 
defenslblllty  using  the  guidelines  Interpretable  from  case  law  on  equity  and 
fairness  In  selection  procedures  and  those  with  low  ratings  were  deleted.  This 
procedure  significantly  reduced  the  pool  of  variables.  This  first  procedure 
reduced  the  pool  of  variables  to  observable  behaviors  and  deleted  demographic 
variables  most  related  to  past  opportunities  and  thus  associated  with  past 
discriminations.  A  consensus  of  panel  judgments  then  selected  the  final 
subset  of  variables  consisting  of  24  rating  elements:  three  overall  ratings 
(job  performance,  supervisory,  managerial);  12  behavioral  ratings  such  as 
responsiveness  to  directions,  self-sufficiency,  etc;  three  ability  ratings 
(quantitative,  reading,  data  Interpretation);  four  motivation  indicators 
(productivity,  Initiative,  speed  of  completion,  and  amount  of  working  time 
spent  In  productive  efforts);  and  two  composite  variables  which  combined 
behavioral  or  motivational  measures.  The  specific  elements  chosen  were  all 
contained  In  the  Job/Worker  Characteristics  Ratings  booklet:  Section  I  Items 
1,  4,  and  5;  Section  III  Items  A  through  L;  Sections  IV  Items  A,  C,  and  H;  and 
Section  VII  Items  1  through  4  (Appendix  B-3  contains  detailed  Item 
descriptions).  Once  the  final  subset  of  24  rating  elements  was  selected,  the 
next  task  was  to  Identify  the  relative  relationship  of  each  element  to  job 
performance  In  each  job  family  and  to  specify  how  those  ratings  were  to  be 
used  In  the  operational  CPAS. 

Promotion  Algorithm  Development 

This  section  describes  the  Promotion  Policy  Data  Panel  (PPDP)  exercise. 
The  purpose  of  the  exercise  was  to  capture  the  promotion  policy  of  a  group  of 
experienced  Federal  service  career-field  representatives,  such  that  their 
policy  (when  converted  Into  a  mathematical  algorithm)  could  be  used  In 
conjunction  with  the  Promotions  and  Placements  Referral  System  (PPRS).  The 
PPRS  Is  a  centralized  computer  system  that  contains  the  current  background, 
experience,  and  rating  data  on  all  appropriated  fund  Air  Force  civilian 
employees  and  Is  used  to  Identify  and  rank  all  ellglbles  for  each  job 
vacancy.  The  CPAS  component  (l.e.,  ratings  and  promotion  algorithm)  would  be 
used  as  the  last  ranking  factor  only  In  cases  where  the  number  of  promotion 
ellglbles  exceeds  the  possibility  for  direct  referral.  To  use  the  CPAS  In 
this  way,  It  was  believed  that  there  must  be  a  common  set  of  rating  factors 
for  all  ellglbles,  even  If  not  all  factors  were  used  for  all  jobs.  Given  this 


purpose,  there  were  three  specific  objectives  of  the  exercise.  The  first 
objective  was  to  provide  restricted  lists  of  rating  elements  that  would  be 
maximally  relevant  to  each  of  the  23  civilian  job  families.  The  second 
objective  was  to  obtain  a  job-famlly-speclflc  weight  for  each  factor,  so  that 
the  factors  might  be  combined  into  a  single  promotion  system,  or 
"promotabillty  score,"  for  each  job  family.  The  third  objective  was  to  obtain 
interim  evidence  for  the  content  validity  of  this  ranking  system,  until  a 
full-scale,  criterion-related  validity  study  could  be  completed  after  the  CPAS 
became  operational. 

After  selection  of  the  workable  subset  of  variables  from  the  data 
collection  Instruments,  some  methodology  had  to  be  used  to  identify  and  weight 
the  specific  promotion  factors  (l.e.,  variables).  Several  techniques  were 
considered  for  possible  use.  However,  in  the  absence  of  an  external  criterion 
of  promotabillty,  the  traditional  multiple  regression-based  technique  was  not 
considered  appropriate  for  the  selection  of  the  final  factors  and  their 
weights.  Another  possibility  considered  was  the  use  of  a  panel  of 
management/personnel  experts  who  could  Identify  the  final  list  of  factors; 
however,  this  technique  would  have  given  no  assurances  about  the  content 
relevance  or  validity  of  the  final  factors.  The  use  of  an  equal-weighting 
scheme,  whereby  all  of  the  original  factors  would  have  been  retained  and  unit 
weighted  was  also  rejected  since  it  would  have  little  face  validity  or 
job-famlly-speclflc  relevance.  After  an  evaluation  of  the  advantages  and 
disadvantages  of  these  techniques,  It  was  decided  that  a  promotion  policy 
panel  exercise  would  provide  the  best  technique  to  solve  and  weight  the  final 
set  of  job-relevant  promotion  factors. 

This  section  describes  the  activities  and  objectives  of  the  Promotion 
Policy  Data  Panel  (PPDP)  exercise.  The  purpose  of  the  exercise  was  to  capture 
mathematically  the  experienced-based  judgments  of  senior  employees  concerning 
the  job-relevance  and  weighting  of  the  rating  elements.  The  selected  research 
strategy  used  "policy-capturing,"  a  statistical  judgmental  analysis  technique 
developed  at  the  Manpower  and  Personnel  Division  (Black,  1973;  Chrlstal, 
1963).  Basically,  the  technique  consisted  of  convening  a  series  of 
job-famlly-speclflc  experimental  promotion  boards  (the  Promotion  Policy  Data 
Panels,  or  PPDPs)  to  perform  two  tasks:  (a)  select  a  list  of  6  to  10 
promotion  factors  for  a  particular  job  family;  and  (b)  rank-order  (sort)  a 
selected  random  sample  of  employees  In  the  job  family  according  to  the 
employees'  promotabillty.  Then  the  policy-capturing  software  Is  used  to 
determine  the  relative  value  that  panel  members  attached  to  the  rating 
elements  In  making  the  promotabillty  rank  orders.  The  result  was  objective 
formulas  or  algorithms  which  replicate  the  ranking  judgments  of  the  panels. 

Promotion  Policy  Data  Panels.  For  panel  composition,  stratified  random 
sampling  techniques  were  used  to  select  participants  to  balance  representation 
by  race  (Black,  Hispanic,  other),  sex,  and  supervlsory/nonsupervlsory  status. 
Selectees  (l.e.,  subject-matter  specialists)  proportionally  represented  job 
series,  locality,  and  major  commands  while  meeting  minimum  grade,  experience, 
performance,  and  availability  requirements. 
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Panels  of  10  to  12  subject-natter  specialists  (SMSs)  were  convened  for  1 
week  in  each  of  the  23  job  families  to  select  the  final  set  of  factors  to  be 
used  for  rating  and  ranking  pronotion  ellgibles  in  their  job  family.  In  two 
job  families,  additional  panels  were  selected  and  convened  to  permit  double 
cross-validation  of  the  panel  variable  selection  process  and  the  promotion 
policies  or  formulas  resulting  from  the  policy-capturing  exercise.  Appendix  C 
details  the  panel  member  selection  process.  Table  C-2  summarizes  the 
characteristics  of  each  panel. 

Variable  Selection  Process.  Each  panel's  first  task  was  to  review  and 
discuss  the  24  prospective  rating  elements.  Next,  they  had  to  select  a  set  of 
6  to  10  elements  which  were  observable  on  the  job  in  their  job  family  and  were 
perceived  as  related  to  ability  to  perform  at  the  next-higher-level 
nonsupervlsory  job.  The  process  was  repeated  to  select  a  second  set  of  6  to 
10  elements  for  supervisory  jobs.  Panels  were  limited  to  10  factors  since  the 
developers'  past  experience  with  policy-capturing  indicates  that  members  can 
not  effectively  consider  more  than  10  factors.  All  variables  except  one. 
Arithmetic  Computation,  were  selected  for  use  by  at  least  one  panel. 

Experimental  Promotion  Board  Activity.  The  second  task  of  the  panel  was 
to  convene  as  a  civilian  promotion  board  to  rank-order  promotion-eligible 
candidates  based  on  the  rating  data  provided.  Ratings  for  each 
promotion-eligible  candidate  were  contained  on  one  data  card  and  reflected  the 
actual  ratings  collected  In  the  CONUS-wlde  field  test.  The  rating  factors  for 
any  specific  candidate  being  ranked  were  only  those  that  had  been  previously 
selected  by  the  panels  for  determining  promotablllty  In  their  particular  job 
family. 

Each  panel  member  was  given  9  to  15  decks  consisting  of  70  to  80  cards 
representing  possible  promotion-eligible  employees.  Panel  members  were  also 
given  copies  of  the  Job/Worker  Characteristics  Ratings  booklet,  from  which  the 
rating  elements  had  been  taken,  and  a  summary  layout  for  each  data  deck. 
Decks  were  labeled  according  to  the  specific  grade  level  and 
supervlsory/nonsupervlsory  nature  of  the  candidate  job  for  which  each 
promotion  eligible  was  being  considered.  Identifying  Information  such  as 
name,  race,  sex,  or  specific  experience  was  not  Included  on  the  data  cards,  to 
provide  anonymity  of  the  field  test  participants.  Each  panel  was  then  asked 
to  rank-order  the  data  cards  (representing  promotion  ellgibles)  according  to 
their  promotablllty  or  potential  to  perform  In  the  next-higher-level  job. 

Panel  members  ranking  wage  grade  job  families  considered  only 
nonsupervlsory  jobs,  whereas  the  panel  ranking  the  wage  supervisory  family 
considered  both  supervisory  and  nonsupervlsory  jobs.  The  single  job  type 
rankings  for  wage  grade  jobs  account  for  the  small  number  of  decks  considered 
by  some  panels. 

Unknown  to  the  panel  participants,  one  duplicate  deck  of  cards  was 
Included  In  each  panel  In  order  to  Identify  inconsistent  raters.  Based  on  the 
analysis  of  the  duplicate  decks,  ranking  data  from  any  Inconsistent  raters 
which  might  bias  the  results  of  the  study  could  be  omitted  for  further  data 
analysis. 
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The  ranking  exercise  required  a  maximum  of  4  days  for  each  job  family. 
Data  Analyses 


Analyses  of  the  Promotion  Policy  Data  Panels  had  several  questions  to 
answer.  Was  there  adequate  Intra-  and  Inter-rater  reliability;  that  Is,  did 
the  panel  members  have  a  consistent  policy  when  ranking  Individuals  on 
promotablllty  and  did  their  policy  agree  with  those  of  other  panel  members? 
Was  it  possible  to  reproduce  mathematically  the  panels'  rankings  such  that 
additional  promotion  ellglbles  could  be  rank  ordered  the  same  as  the  panel 
would  rank  them,  without  active  panel  Input?  Would  the  panel  policies  have 
been  different  If  another  group  of  experts  had  been  used?  If  the  mathematical 
formula  or  algorithm  derived  was  successful  In  weighting  the  rating  factors 
and  reproducing  the  board  rankings,  were  separate  algorithms  required  for 
every  grade  level  for  every  type  of  job  (supervisory  or  nonsupervlsory)  in 
every  job  family?  Finally,  could  the  algorithms  be  simplified  without  losing 
their  predictive  efficiency?  Although  all  these  questions  must  be  addressed, 
it  was  particularly  Important  that  answers  to  the  questions  of  consistency  In 
ratings  be  verified;  otherwise,  the  further  development  of  CPAS  for 
operational  use  would  be  useless. 

Consistency  Analyses.  As  previously  Indicated,  two  Identical  decks  were 
included  among  the  set  of  decks  ranked  by  each  panel  member  to  check  panel 
member  consistency.  (During  the  panel  activities,  none  of  the  panel  members 
Indicated  that  they  detected  the  duplicate  decks.)  The  consistency  of  the 
rankings  assigned  to  cases  In  the  duplicate  decks  was  checked  for  each  panel 
member  by  computing  Pearson  product-moment  correlation  coefficients  for  the 
two  sets  of  rankings.  It  was  arbitrarily  decided  that  .80  was  the  acceptable 
cutoff  for  Intra-rater  consistency.  Only  one  panel  member  had  a  correlation 
of  .68,  which  fell  below  the  preset  cutoff.  This  panel  member's  rankings  were 
not  Included  In  future  analyses.  The  Intra-rater  level  of  consistency  was 
very  high;  91  percent  of  the  panel  members  had  correlations  that  were  .95  or 
greater.  Table  C-4  In  Appendix  C  gives  a  full  breakout  of  these  correlations. 

Evaluating  consistency  among  panel  members  to  see  If  a  panel  consensus 
existed  or  If  there  were  aberrant  panel  members  was  more  complex.  Since  the 
panels  provided  data  both  for  a  nonsupervlsory  and  a  supervisory  policy  for 
most  job  families,  separate  analyses  were  accomplished  using  non-supervlsory 
and  then  supervisory  data.  The  procedure  consisted  of  first  developing  a 
separate  policy  equation  for  each  panel  member  and  then  Inputting  each 
respective  set  of  policy  equations  to  a  hierarchical  grouping  analysis  to 
determine  which  panel  members  had  similar  policies.  Regression  analysis  was 
used  to  develop  the  Individual  members'  equations.  Rank  order  assigned  to  the 
data  cards  (or  cases)  was  the  criterion  measure.  The  predictive  efficiency  of 
a  regression  equation  was  evaluated  In  terms  of  the  multiple  correlation 
coefficient.  Appendix  C  describes  the  regression  weighting  process. 

A  multiple  correlation  of  .90  was  preset  as  the  acceptable  minimum  value. 
Only  one  panel  member  had  an  Individual  policy  equation  (non-supervlsory)  with 
predictive  efficiency  below  the  cutoff  (Rz».81).  This  member's  data  were 


removed  from  further  analysis.  Eighty  percent  of  the  squared  multiple 
correlations  were  .95  or  greater.  Appendix  C  (Tables  C-5  and  06)  gives  the 
full  listing  of  values. 


The  last  and  Most  crucial  consistency  analysis  was  developing  a  group  or 
panel  consensus  equation.  The  hierarchical  grouping  process  used  for  this 
purpose  Is  described  In  detail  in  Appendix  C.  Basically,  hlerarchlal  grouping 
starts  with  two  panel  members  and  develops  a  single  regression  equation  to 
predict  the  rank  orders  each  assigned  and  compares  the  predictive  efficiency 
of  the  joint  equation  to  the  efficiencies  of  their  respective  Individual 
equations.  Then  a  third  rater  Is  added  to  the  group  and  the  change  In 
predictive  efficiency  Is  evaluated  and  so  on  until  the  Iterative  process  has 
added  all  the  raters  to  a  single  group.  Tests  of  statistical  significance  are 
used  at  each  Iteration  to  see  If  there  was  a  significant  loss  in  predictive 
efficiency  at  that  stage  which  would  Indicate  that  a  panel  member's  policy 
differed  from  the  other  members.  It  was  found  that  none  of  the  panel  members' 
Individual  policy  equations  substantially  disagreed  with  those  of  the  other 
panel  members  and  It  was  feasible  to  compute  single  panel  equations.  The 
lowest  multiple  correlation  from  the  hierarchical  grouping  process  was  .93. 
Based  on  the  results  obtained  from  these  analyses,  it  was  concluded  that  panel 
consensus  had  been  reached,  the  panel  equations  were  highly  predictive,  and 
the  computation  of  the  final  CPAS  algorithms  could  begin. 


Computation  of  the  Promotion  Policy  Equations.  Several  analyses  were 
performed  In  arriving  at  the  final  promotion  policy  equations.  Basically,  the 
hierarchical  regression  grouping  process  just  described  was  used  to  evaluate 
similarities  In  policy  equations  across  grade  levels,  and  across  grade  levels 
within  nonsupervlsory  and  supervisory  jobs.  Results  of  these  analyses 
Indicated  that  there  were  no  significant  grade-level  category  differences 
within  job-family  policies,  so  a  single  equation  could  be  used  within  a 
particular  job  family  without  a  significant  loss  in  predictive  efficiency. 
Predicted  scores  from  the  combined  grade-level  equations  showed  very  high 
agreement  with  those  resulting  from  the  separate  grade-level  equations.  The 
correlations  between  the  two  sets  of  promotion  scores  based  on  the  combined 
and  separate  equations  were  all  .98  or  higher.  The  magnitude  of  the 
correlations  Indicated  that  the  panels'  grade-level  rankings  could  be 
reproduced  with  exceptional  accuracy  by  the  non-graded  algorithm  weights  for 
each  job  family. 


Significant  differences,  however,  were  found  between  supervisory  and 
nonsupervlsory  panel  policies  within  and  between  job  families.  This 
necessitated  the  use  of  separate  supervisory  and  nonsupervlsory  policy 
equations  (algorithms)  for  each  job  family  having  both  job  types.  Therefore, 
a  total  of  40  CPAS  algorithms  were  required  for  the  23  job  families: 
supervisory  and  nonsupervlsory  algorithms  for  job  families  1  through  17,  only 
a  supervisory  algorithm  for  job  family  18,  and  only  nonsupervlsory  algorithms 
for  job  families  19  through  23.  Specific  analyses  for  the  development  of 
these  algorithms  are  detailed  In  Appendix  C. 


Identification  of  Operational  Weighting  Systems.  The  next  analysis  task 
was  algorithm  simplification.  Since  It  was  noted  that  the  weights  for  two 
factors  (data  Interpretation  and  reading  ability)  had  near-zero  values  In  Most 
algorithms,  they  were  removed  from  the  algorithms  and  the  Impact  of  their 
elimination  evaluated.  Results  Indicated  that  there  was  no  significant  effect 
on  the  predictive  efficiency  of  the  algorithms  using  only  the  remaining 
factors.  This  process  reduced  the  total  number  of  factors  to  be  used  In  the 
operational  CPAS  to  19. 


Some  of  the  remaining  factor  weights  were  very  small  (near  zero)  In 
selected  algorithms,  a  few  were  negative,  and  all  contained  values  computed  to 
the  sixth  decimal  place.  Past  experience  with  weighting  promotion  factors 
suggested  that  the  weights  could  be  simplified  by  changing  them  to  whole 
numbers  without  adversely  affecting  the  prediction  system  (Black,  1973). 
Since  both  the  computation  and  explanation  of  the  weights  used  In  the 
algorithms  would  be  simplified  by  use  of  Integer  weights,  a  decision  was  made 
that  all  weights  would  be  rounded  to  the  nearest  whole  number  and  weights  less 
than  "I*  would  be  set  to  zero.  The  new  Integer  weights  were  used  to  generate 
algorithm  scores;  then  the  algorithm  scores  were  correlated  with  averaged 
panel  percentile  ranks  to  assess  the  efficiency  In  using  the  new  weights.  As 
shown  In  Table  1,  correlations  ranged  from  .92  to  .98,  Indicating  that  the  use 
of  the  simplified  weights  would  not  decrease  the  overall  predictive  efficiency 
of  the  system.  Therefore,  It  was  decided  that  the  simplified  weights  would  be 
used  In  the  operational  CPAS.  Additional  analyses  comparing  the  operational 
CPAS  algorithm  rankings  with  the  panel  rankings  are  presented  In  Appendix  C. 


The  operational  algorithms  generate  an  employee's  CPAS  score  by  applying 
the  appropriate  job-family  Integer  weighting  systems  (nonsupervlsory  or 
supervisory)  to  the  employee's  score  on  the  respective  CPAS  promotion 
factors.  For  example,  the  CPAS  score  for  an  employee  In  a  job  family  with 
five  non-zero  weighted  factors  would  be: 


CPAS  Score  «  (F^i  X  Fr])  +  (F^  X  F^)  +  (Fw3  X  Fr 3 )  + 
(Fw4  x  Fr4)  +  (Fw5  X  Fr5) 


where  Fw  represents  the  Integer  weight  for  a  factor,  and 
Fr  represents  the  factor  rating  given  the  employee  In  the 
CPAS  rating 
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Table  1.  Correlations  Between  Operational  CPAS 
Algorithm  Scores  and  Composite  Panel  Ranks 


Job  Family 


Nonsupervisory  Supervisory 


N 

r 

N 

r 

1 

714 

.940 

256 

.948 

2 

320 

.951 

167 

.953 

3 

513 

.943 

189 

.972 

4 

545 

.944 

324 

.953 

5 

460 

.940 

206 

.953 

6 

555 

.943 

198 

.950 

7 

353 

.936 

141 

.937 

8 

650 

.949 

278 

.962 

9 

587 

.965 

214 

.982 

10 

574 

.959 

315 

.980 

11 

700 

.951 

229 

.963 

12 

422 

.953 

225 

.953 

13 

543 

.936 

283 

.951 

14 

439 

.946 

315 

.937 

15 

576 

.960 

209 

.972 

16 

640 

.968 

162 

.981 

17 

726 

.919 

114 

.958 

18 

N/A* 

N/A* 

232 

.933 

19 

669 

.966 

N/A* 

20 

488 

.959 

N/A 

21 

545 

.957 

N/A 

22 

480 

.974 

N/A 

23 

565 

.971 

N/A 

*  N/A  »  Not  Applicable. 


The  factor  weights  are  whole  integers  and  range  from  1  to  10  for  the 
nonsupervisory  and  1  to  11  for  the  supervisory  weighting  systems.  Appendix  C 
(Tables  C- 1 7  and  C- 18 )  displays  the  possible  point  ranges  for  the 
nonsupervisory  and  supervisory  CPAS  weighting  systems,  along  with  the  CPAS 
score  ranges,  means,  and  standard  deviations  for  the  job-family  sample  cases 
used  In  the  development  of  the  respective  weighting  systems.  The  Directorate 
C'  Civilian  Personnel  has  declared  that  the  actual  promotion  weights  are 
"Privileged  Information*  and  thus  cannot  be  included  in  the  body  of  the 
report.  Similarly,  the  specific  factors  weighted  for  each  job  family  are  not 
Identified. 


Cross-Validation  of  CPAS  Algorithms.  To  cross-valldate  the  panel  process 
(content  validation)  and  the  derivation  of  the  operational  algorithms 
(construct  validation),  two  additional  promotion  policy  data  panels  were 
convened  for  Job  Family  11,  Business  Technical,  and  Job  Family  20,  General 
Mechanical.  The  panel  process  and  algorithm  development  process  for  each  was 
the  same  as  described  previously.  The  within- job-family  replication  panels 
were  convened  at  different  times  and  did  not  know  of  each  other's  existence. 
Results  from  the  comparative  data  Indicated  the  rank  orders  assigned  to 
nonsupervisory  and  supervisory  cases  for  Family  11  and  nonsupervlsory  cases 
for  Family  20  were  virtually  Identical  for  the  wlthln-famlly  algorithms 
developed  for  each  panel.  Correlations  between  the  two  panel  rankings  were 
.94  to  .98  for  Family  11  and  Family  20  showing  remarkable  consistency  (l.e., 
content  and  construct  validity)  for  the  Family  11  and  Family  20  algorithms. 
Based  on  the  analyses  for  these  two  job  families,  there  Is  no  reason  to 
believe  that  such  consistency  would  not  be  found  for  the  remaining  job-family 
algorithms. 


III.  DISCUSSION 


Implementation  Issues 


Three  basic  Implementation  Issues  needed  to  be  addressed.  They  dealt  with 
development  of  the  operational  rating  form,  procedures  for  managing  rating 
Inflation,  and  training  of  management,  supervisors,  and  employees. 


Operational  Rating  Form 


In  their  original  request,  USAF/MPK  requested  simplicity  In  the  rating 
forms.  It  was  also  recognized  that  the  rating  forms  were  to  be  used  In  an 
automatic  data  processing  (ADP)  environment.  Since  different  job  families 
used  different  combinations  of  the  19  rating  factors,  the  advisability  of 
having  separate  rating  forms  for  each  job  family  was  considered.  Separate 
family  forms  would  reduce  to  a  minimum  the  number  of  rating  factors  that 
supervisors  would  have  to  use  hut  would  result  In  the  need  for  23  separate 
rating  forms.  By  combining  job  families  using  similar  factors,  only  seven 
rating  forms  would  be  required,  thus  significantly  reducing  the  total  number 
of  forms  to  a  more  manageable  number. 
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To  provide  empirical  data  related  to  this  Issue  and  before  a  final 
decision  was  Made  on  the  nuiiber  of  foras,  AFHRL  requested  that  OCPO  conduct  an 
analysis  of  eaployee  aoveaent  to  see  If  employees  tended  to  remain  within  a 
certain  Job  faally.  Fro*  their  historical  data  base,  OCPO  constructed  a 
■atrlx  of  eaployee  aoveaent  within  and  between  job  families.  Surprisingly, 
aore  Individuals  were  proaoted  froa  outside  their  job  faally  than  within. 
Such  aoveaent  between  job  faallles  aeant  that  Individuals  who  were  rated  on 
one  fora  would  have  to  be  rated  on  additional  forms  when  they  wanted  to 
coapete  for  proaotlon  within  other  job  faallles.  Considering  the  results  of 
the  aoveaent  analysis,  rating  requlreaents  for  Multiple  rating  forms  would  be 
substantial  and  probably  unaanageable. 


To  achieve  the  desired  slapllclty,  AFHRL  recommended  that  a  universal 
rating  fora  be  used  whereby  all  employees  would  be  rated  on  all  19  factors 
regardless  of  their  applicability  to  proaotlon  In  the  eaployee*s  current  job 
faally.  Based  on  this  recoaaendatlon,  the  Director  of  Civilian  Personnel 
decided  that  a  universal  rating  fora  would  be  used  for  the  operational 
system.  This  decision  had  the  added  advantage  that  now  every  Individual  could 
be  considered  for  proaotlon  for  every  job  vacancy  for  which  that  person  was 
eligible,  without  delay.  With  the  use  of  the  Promotions  and  Placements 
Referral  System  (PPRS)  within  the  centralized  data  systea,  eligible  candidates 
could  even  be  Immediately  considered  for  all  CONUS  job  openings  regardless  of 
their  base  of  assignment. 


In  their  original  fora  In  the  Job/Worker  Characteristics  Ratings  booklet, 
the  19  selected  factors  were  operationally  defined  such  that  all  persons  In 
all  jobs  should  have  observable  job  components  relating  to  some  facet  of  each 
factor.  These  factors  also  had  generic  factor  names  such  as  Leadership, 
Skill,  Supervisory  Ability  and  Management  Ability. 


It  was  recoaaended  by  AFHRL  that  these  factor  names  be  removed  from  the 
operational  rating  form  for  four  reasons.  First,  raters  might  respond  to  the 
factor  name  without  reading  the  operational  definition  and  render  ratings 
based  on  their  own  deflnlt'ons  of  leadership,  rather  than  having  a  frame  of 
reference  common  to  all  raters.  Second,  some  raters  might  resist  rating  some 
factors,  based  on  the  factor  name  alone.  For  example,  a  supervisor  might 
hesitate  to  rate  a  non-supervlsory  eaployee  on  supervisory  ability.  However, 
the  operational  definition  of  supervisory  ability  describes  behaviors  which 
are  related  to  supervisory  activities  but  which  do  not  necessarily  have  to  be 
performed  In  a  supervisory  mode.  Third,  It  was  expected  that  raters  would  be 
more  likely  to  avoid  Inducing  halo  effects  Into  their  ratings  if  they  reacted 
to  operational  definitions  rather  than  factor  names.  Finally,  and  most 
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Importantly,  the  Uniform  Guidelines  (1976)  and  case  law  (Casclo  &  Bernardln, 
1981)  Indicate  that  rating  Individuals  on  traits  and  characteristics  was  not 
acceptable.  Instead,  ratings  should  be  In  terms  of  behavioral  dimensions. 

Removing  the  generic  factor  names  removes  the  temptation  to  rate  the 
factors  as  traits  or  characteristics  and  fosters  the  use  of  the  rating 
elements  as  behavioral  dimensions. 

The  CPAS  rating  form  (AF  Form  1287)  Is  reproduced  In  Appendix  D.  As 
recommended  by  AFHRL,  9-polnt  normative  rating  scales  were  used.  The  specific 
format  or  layout  of  the  form  was  adopted  by  a  CPAS  Implementation  Working 
Group,  which  consisted  of  representatives  of  each  major  command  Director  of 
Personnel  (DP)  and  OCPO. 

Inflation  Management 

At  the  outset,  USAF/MPK  wanted  a  rating  system  that  was  resistant  to 
Inflation.  Although  It  was  not  believed  that  an  entirely  Inflation-free 
rating  system  could  be  developed.  It  was  deemed  possible  to  manage  Inflation 
such  that  there  would  be  sufficient  variance  to  permit  accurate  ranking  of 
employees.  Past  research  Indicates  that  attempting  to  obtain  variance  by 
normalizing  or  curve-forcing  actions  or  by  very  stringent  rater-training 
programs  does  produce  variance  In  ratings  but  that  much  of  It  Is  error 
variance  rather  than  reflecting  any  true  differences  (Hedge,  1983). 
Therefore,  the  objective  was  to  "manage"  rather  than  "eliminate"  rating 
Inflation  through  a  system  that  fostered  Information  flow  to  managers  and 
supervisors  and  created  realism  In  ratings. 

Four  specific  actions  were  recommended  to  promote  Inflation  management  of 
the  rating  system.  These  recommendations  focused  on  the  format  of  the  rating 
scale,  the  advisability  of  rating  all  employees  at  one  time  In  a  unit,  the 
need  for  separate  job  performance  and  promotion  rating  systems,  and  the 
utility  of  a  feedback  system. 

It  was  believed  that  the  most  effective  rating  scale  would  be  a  9-polnt 
behavlorally  anchored  scale  where  the  rated  Individual  would  be  compared  to 
the  average  employee.  The  average  employee  should  be  defined  as  being  highly 
effective  and  motivated  In  the  perforitance  of  the  assigned  job.  Ratings  using 
the  top  or  bottom  two  scale  points  should  be  substantiated  by  citing  specific 
performance  events  relevant  to  that  rating.  This  rating  scale  format  was 
approved  by  USAF/MPK,  and  Instructions  for  justifying  extreme  ratings  and 
completion  and  Interpretation  of  the  rating  summary  work  sheets  were  outlined 
In  a  brochure  developed  by  OCPO  with  AFHRL  Input.  A  copy  of  the  CPAS 
Supervisory  Brochure  Is  Included  In  Appendix  E. 

The  AFHRL  developers  believed  that  a  major  reason  for  Inflation  In  the 
then-existing  promotion  rating  system  was  that  ratings  were  rendered  on 
employees  at  differing  times.  This  meant  that  supervisors  and  management 
could  not  compare  employees  within  an  organization  at  the  same  time.  Without 
a  visible  comparison,  there  was  no  way  to  ensure  that  the  proper  distinctions 
were  being  made  between  employees  and  that  the  ratings  of  a  particular 
supervisor  or  unit  were  not  overinflated.  By  rating  all  Individuals  In  a  unit 
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simultaneously,  raters  should  be  able  to  make  more  objective  comparisons  among 
subordinates.  Each  supervisor's  ratings  should  be  summarized  on  work  sheets 
and  the  summaries  submitted  with  the  ratings  to  an  Indorsing  official. 
Indorsing  officials  can  then  check  for  rating  errors  such  as  leniency,  halo, 
or  response  set.  Further,  the  work  sheet  summaries  will  permit  the  Indorsers 
to  make  comparisons  across  raters  and  ensure  that  ratings  are  consistent  with 
each  unit's  productivity.  Indorsers  should  be  authorized  to  mandate  that 
ratings  be  consistent  with  actual  performance. 

As  recognized  by  USAF/MPK,  when  an  appraisal  Instrument  Is  used  for  a 
variety  of  purposes.  It  tends  to  make  the  Instrument  Ineffective  for  any 
purpose.  It  appears  that  the  greater  the  number  of  uses  for  a  rating,  the 
greater  the  pressure  on  the  supervisor  to  Inflate  the  rating.  For  this 
reason,  AFHRL  recommended  that  the  CPAS  ratings  be  separated  from  the  job 
performance  rating  system  entirely.  The  only  relationship  between  the  two 
systems  should  be  (a)  that  Individuals  must  have  obtained  at  least 
satisfactory  job  performance  ratings  (l.e.,  under  the  General  Manager 
Appraisal  System  (GMAS)  or  the  Job  Performance  Appraisal  System  (JPAS) )  to  be 
promotion  eligible,  and  (b)  that  the  ability  to  render  appropriate  CPAS 
ratings  should  be  a  mandatory  component  of  each  supervisor's  GMAS  or  JPAS 
rating.  Separation  of  the  two  systems  was  approved  and  has  been  operationally 
Implemented. 

When  rating  feedback  systems  are  used,  the  system  usually  provides  mean 
summary  values  of  ratings  so  that  comparisons  among  raters  and  comparisons  of 
subordinate  units'  rating  policies  can  be  made.  Anecdotal  Information 
suggests  that  when  such  a  system  Is  used,  organizations  with  lower  mean 
ratings  feel  they  may  have  been  unfair  to  their  employees  and  Increase  the 
average  of  their  next  ratings.  Although  It  was  recognized  that  a  feedback 
system  may  actually  Increase  rather  than  decrease  Inflation,  such  a  system  was 
still  recommended  for  Implementation  of  CPAS.  Kith  CPAS,  however,  the 
recommended  system  was  to  provide  only  variances  In  ratings  given,  not  mean 
rating  values.  Accompanying  guidance  would  point  out  that  "the  more  variance, 
the  better*  would  be  the  preferred  policy.  Since  Inflation  reduces  score 
dispersion,  hence  the  amount  of  variance,  the  resulting  lack  of  variance  would 
be  an  Indication  of  poor  rating  management  and  Inability  of  the  supervisor  to 
differentiate  performance  between  Individuals.  This  variance  yardstick  could 
then  be  used  to  guide  subsequent  performance  ratings  of  supervisors.  The  job 
performance  rating  systems  now  used  by  the  Air  Force  (GMAS  and  JPAS),  require 
that  all  supervisors  have  a  critical  supervisory  ability  job  element. 
Therefore,  all  supervisors  should  be  rated  on  their  ability  to  rate.  Variance 
feedback  should  make  supervisors  accountable  for  their  rating  styles. 
Although  the  suggested  feedback  system  was  not  Implemented,  It  Is  believed 
that  reconsideration  of  the  recommendation  Is  warranted. 

Training  and  Information  Dissemination 


Experience  with  SEAS,  GMAS,  and  JPAS  Implementation  Indicated  that  those 
who  most  vigorously  objected  to  the  systems  were  those  who  had  not  undergone 
training.  Since  CPAS  was  a  significant  departure  from  the  format  of  ths 
existing  promotion  rating  system,  an  aggressive  training  program  was 
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recommended.  Host  Important  was  the  necessity  to  overcome  the  prevailing 
negative  attitude  toward  promotion  ratings  and  the  concept  that  all  but  the 
very  worst  performers  should  receive  top  ratings.  After  Initial  training, 
periodic  refresher  training  should  be  performed,  possibly  In  conjunction  with 
GMAS  or  JPAS  refresher  training.  A  continuing  training  program  must  be  used 
to  train  new  supervisors,  particularly  military  members,  since  the  military 
rating  system  has  become  highly  Inflated  over  the  years. 

The  CPAS  Implementation  Working  Group  focused  on  what  was  needed  to 
implement  CPAS  successfully.  The  working  group  recognized  the  need  for 
adequate  training  and  the  local  commander's  support  to  make  CPAS  work.  Four 
basic  recommendations  were  made  by  the  group.  A  mandatory  training  program  (1 
day  for  supervisors  and  1/2  day  for  employees)  should  be  Implemented.  The 
training  should  be  professionally  developed  and  should  use  an  audiovisual 
format.  Training  should  be  conducted  well  before  CPAS  implementation  and  have 
formal  traln-the-tralner  sessions. 

To  obtain  the  necessary  support  of  the  commanders,  the  working  group  felt 
a  publicity  campaign  consisting  of  articles  and  briefings  should  be  developed 
by  the  Public  Affairs  Unit  at  Kelly  AFB.  Then  a  team  could  take  the  briefings 
to  the  Air  Staff  and  major  commands. 

Based  on  the  union-induced  delays  during  the  CONUS-wlde  field  test  and 
announced  AFGE  opposition  to  any  promotion  system  other  than  a  seniority 
system,  the  group  recommended  early  union  notification  and  submission  of 
negotiation  guidance  packages  to  the  chiefs  of  civilian  personnel  offices 
(CCPOs)  throughout  the  Air  Force. 

Finally,  the  group  felt  that  a  comprehensive  Informational  package  should 
be  developed  which  explained  CPAS,  Its  origin,  development  process,  and 
mechanics.  The  package  should  then  be  distributed  to  all  employees  who  are 
affected  by  the  system. 

Research  Plan  and  Product  Changes 

A  variety  of  events  changed  the  original  proposed  research  plan  and 
products  to  be  delivered  to  the  personnel  community.  Passage  of  CSRA-78  was 
the  major  event  affecting  CPAS  development.  Shifting  of  scientific  resources 
to  develop  these  snort-suspense  job  performance  appraisal  systems  necessitated 
a  realignment  of  manpower  resources  and  change  In  research  milestones. 

!r.  FY80,  the  exploratory  development  (6.2)  program  element  used  tc  Tund 
the  appraisal  research  and  development  (R&D)  efforts  was  cut  by  Congress  with 
prejudice  because  R&D  funds  were  being  used  by  all  the  services  to  Implement 
CSRA.  Since  the  cut  with  prejudice  meant  the  Air  Force  could  not  reprogram 
funds,  portions  of  the  CPAS  development  program  had  to  be  omitted. 
Specifically,  the  operational  test  and  evaluation  (OT&E)  of  the  finalized 
Cr  AS,  cost-benefits  analyses  comparing  CPAS  to  the  current  system,  and  planned 
publicity  packages  and  the  CPAS  training  system  were  removed  from  the  AFHRL 
list  of  products  to  be  delivered  to  USAF/HPK. 
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USAF/MPK  eliminated  the  use  of  aptitude  tests  as  a  component  of  CPAS 
because  their  use  may  not  be  legally  defensible.  Loss  of  the  CPE  had  the 
additional  effect  of  eliminating  the  use  of  Ipsatlve  rankings  In  CPAS.  The 
CPE  scores  and  supervisory  ratings  on  CPE-measured  dimensions  were  going  to  be 
used  to  help  benchmark  and  normalize  the  Ipsatlve  ratings.  Ipsatlve  ratings 
were  expected  to  add  an  Inflatlon-reslstent  component  to  CPAS. 

A  brief  field  test  of  the  operational  CPAS  was  planned  to  compensate  for 
the  loss  of  the  OT&E.  The  test  was  held  In  conjunction  with  a  final  test  of 
JPAS.  Some  1,000  supervisors  and  employees  were  trained  at  five  bases  In  July 
1981  (Guerrlerl,  Thompson,  Cowan,  &  Gould,  1984).  JPAS  ratings  were  rendered 
In  October  19SI,  and  CPAS  ratings  were  collected,  Including  followup 
Interviews,  In  December  1981.  However,  since  the  field  test  was  made 
voluntary,  so  few  CPAS  ratings  were  obtained  that  no  definitive  analyses  could 
be  made.  The  few  CPAS  respondents  (less  than  100)  appeared  to  favor  CPAS,  but 
no  conclusions  could  be  drawn  since  attitude  toward  CPAS  might  have  Influenced 
the  decision  to  complete  or  not  complete  the  voluntary  CPAS  ratings. 

Legal  Considerations 

As  previously  stated,  consensus  among  Industrial  consultants,  personnel 
specialists,  and  the  AFHRL  research  staff  was  that  any  effective  promotion 
system  would  undergo  many  legal  challenges.  To  make  the  system  legally 
defensible,  the  Air  Force  Central  Labor  Law  Office  at  Randolph  AFB  assigned  a 
legal  advisor  to  the  CPAS  project.  The  specific  contribution  of  the  Law 
Office  was  a  detailed  review  of  the  CPAS  research  and  development  plan  In 
relation  to  the  requirements  of  case  law  and  the  Uniform  Guidelines  for 
Employee  Selection.  Suggested  changes  were  adhered  to,  particularly  In  the 
variable  reduction  process. 

Although  the  Uniform  Guidelines  generally  apply  only  If  a  selection  system 
creates  an  adverse  Impact  on  minorities,  the  Implications  of  the  principles 
expressed  are  good  selection  system  development  practices.  Under  the 
guidelines,  adverse  Impact  can  be  justified  when  the  system  Is  a  valid 
predictor  of  performance  and  specified  development  actions  were  taken. 

As  developed,  CPAS  meets  the  requirements  of  the  guidelines.  The 
suggested  selection  system  development  steps  were  followed,  and  the  system 
development  data  give  strong  evidence  that  CPAS  has  both  content  and  construct 
validity.  Since  the  guidelines  state  that  failure  to  collect  and  maintain 
data  for  assessment  of  possible  adverse  Impact  Is  to  be  considered  as  evidence 
of  adverse  Impact,  the  user  must  now  collect  and  maintain  necessary  records  to 
advert  this  change.  Further  evidence  must  be  collected  to  establish 
predictive  validity  since  the  guidelines  consider  content  and  construct 
validity  to  be  acceptable  only  as  Interim  validity  for  new  systems  until 
predictive  validity  can  be  established. 

IY.  RECOMMENDATIONS 

Recommendations  were  made  to  USAF/MPK  throughout  the  development  of  the 
CPAS;  however,  after  operational  Implementation,  several  additional 
recommendations  should  be  considered  for  the  future. 


It  Is  recommended  that  an  aggressive  ongoing  evaluation  and  analysis 
program  be  established.  The  analysis  program  should  cover  the  entire 
promotion  selection  process,  particularly  the  progression-level  factors  In  the 
Promotion  Placement  and  Referral  System  (PPRS).  CPAS  Is  only  the  last  ranking 
factor  In  PPRS.  Case  law  Indicates  that  even  if  a  system  does  not  create 
adverse  impact  in  Its  total  effect,  the  system  Is  still  considered  to  have 
adverse  Impact  If  such  can  be  demonstrated  for  any  system  subcomponent  (Casclo 
&  Bernardln,  1981). 

Three  basic  areas  of  concern  should  be  addressed  by  the  evaluation 
program:  (a)  that  the  users  are  following  proper  procedures,  (b)  that  CPAS  Is 
effective  In  helping  managers  Identify  the  best  promotion  ellglbles,  and  (c) 
that  CPAS  meets  legal  and  regulatory  requirements.  An  evaluation  and  analysis 
program  should,  as  a  minimum,  perform  the  following  functions: 

1.  Checking  for  adherence  to  CPAS  procedures.  This  area  should  be  an 
action  Item  added  to  self-inspections,  IG  Inspections,  CCPO  audits  of 
personnel  actions,  and  travel  team  evaluations.  The  objectives  should  be  to 
ensure  that  there  is  a  continuing  program  of  Initial  and  refresher  training 
for  supervisors,  that  current  appraisals  are  being  used,  and  that  the  proper 
CPAS  algorithms  are  used  In  specific  fill  actions. 

2.  Monitoring  the  effectiveness  of  the  CPAS  program.  Efforts  should  be 
directed  on  a  periodic  basis  to  review  rating  trends  of  the  program  and.  If 
rising  Inflation  Is  apparent,  to  propose  corrective  measures  to  Improve  the 
rating  process.  In  addition,  attention  should  be  directed  to  the  supervisors 
and  employees  participating  In  the  system  and  their  views  toward  the  overall 
program.  Surveys  and  questionnaires  should  be  used  to  provide  Indications  of 
acceptability  to  employees  and  supervisors. 

3.  Ensuring  that  CPAS  meets  legal  and  regulatory  requirements.  The 
basic  steps  needed  to  maximize  the  chance  for  successful  defense  against  legal 
challenges  would  be  to  track  adverse  Impact,  document  system  changes,  and 
further,  establish  CPAS  predictive  validity.  Adverse  Impact  Is  meaningful 
only  In  terms  of  actions  taken  rather  than  In  terms  uf  differences  In  minority 
versus  majority  CPAS  scores.  The  numbers  and  classifications  of  eligible 
candidates  who  are  rank-ordered  by  CPAS  should  be  compared  to  the 
characteristics  of  groups  actually  referred  for  interview.  This  Is  the  only 
appropriate  evaluation  of  CPAS  for  adverse  Impact.  Similarly,  each 
progression  level  In  PPRS  should  be  evaluated  on  the  same  basis. 

CPAS  system  changes  such  as  adding  new  job  series  to  job  families  or  using 
CPAS  for  a  new  purpose,  such  as  cadre  selection  for  careers  management 
programs,  should  be  well  documented.  Finally,  external  job  performance 
measures  should  be  established  and  CPAS  scores  correlated  with  these  measures 
to  assess  the  predictive  validity.  The  process  will  be  complex  and 
time-consuming  because  It  will  be  a  long  time  before  sufficient  promotions 
will  occur  In  a  job  family  to  permit  predictive  analyses.  In  the  Interim, 
concurrent  validities  comparing  CPAS  scores  with  current  performance  measures, 
such  as  GMAS  and  OPAS,  should  be  conducted. 
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APPENDIX  A:  JOB  FAMILY  SPECIFICATIONS 


General  Charter 


Since  the  development  of  the  CPAS  was  dependent  on  an  appropriate  grouping 
of  approximately  1,500  civilian  job  series  Into  a  manageable  number  of  logical 
and  homogeneous  clusters  of  job  types,  a  working  group  consisting  of  position 
classifiers  and  research  psychologists  was  established  and  tasked  to  develop 
these  homogeneous  clusters.  The  following  paragraphs  describe  the  panel 
process  by  which  the  final  job  groupings  were  derived. 

The  following  working  group  charter  was  agreed  to  by  AFHRL  and  OCPO. 

The  objective  of  this  work  group  Is  to  assign  occupational  groups 
and  job  families  to  categories  that  support  research  requirements. 

The  work  group  will  review  categories  previously  developed  by  AFHRL 
and  OCPO  as  well  as  the  entire  list  of  occupational  groups  and  job 
families  published  by  0PM  or  presently  In  use  within  the  Air 
Force.  Categories  developed  by  the  work  group  must  represent 
sufficient  personnel  to  support  an  appraisal  algorithm  and  have 
homogeneous  qualities  relative  to  attributes  required  In  the  job 
(l.e.,  skills,  knowledge,  experience,  and  training).  The  work 
group  will  reduce  the  number  of  categories  to  the  smallest  possible 
total  within  the  guidelines  listed  below.  The  work  group  will  be 
divided  Into  two  panels,  each  arriving  at  Its  own  conclusions  on 
categories  of  occupational  groups  and  job  families.  Each  panel 
will  assign  a  rating  to  factors  for  the  occupational  groups  and  job 
families  considered  for  Inclusion  in  a  category.  These  ratings  are 
later  to  be  compared  statistically  to  determine  the  agreement 
between  the  two  panels  and  the  adequacy  of  the  criteria  provided. 

The  panels  will  keep  a  detailed  record  of  all  judgments  made  (e.g., 
rationale  behind  use  of  additional  criteria  during  selection  of  an 
occupational  group  or  job  family  for  a  category),  so  problem  areas 
can  be  clearly  Identified  and  resolved,  and  so  a  complete  history 
Is  available  for  analysis  and  future  application  or  modification  of 
procedures.  The  work  group  will  submit  the  results  of  their 
efforts  In  writing  to  OCPO/MPKEA  who  will  provide  them  to  AFHRL. 

The  Chairman  of  the  group  will  be  the  OCPO/MPKMM  representative. 

Working  Group  Report  Summary 


The  following  paragraphs  summarize  the  working  group  report  and  resulting  job 
family  specifications. 

1.  Working  group  composition.  As  a  preliminary  step  In  Implementing  the 
performance  appraisal  process,  a  work  group  was  formed  to  divide  the  Air  Force 
occupations  Into  families.  These  families  would  be  made  up  of  series  with 
tasks  that  are  similar  enough  to  be  appalsed  by  like  methods.  The  work  group 
composition  Is  as  follows: 
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Lynn  Carpenter,  OCPO/MPKMM,  Position  Classification  Specialist,  Chair 

Capt  William  Cunnings,  AFHRL,  Behavioral  Scientist 

Dwight  Hall,  OPO/MPKEE,  Personnel  Management  Specialist 

Ron  Sheppard,  OCPO/HPKT,  Personnel  Management  Specialist 

Bob  Utterback,  OCPO/MPKCL,  Personnel  Management  Specialist 

James  Wllbourn,  AFHRL,  Personnel  Research  Psychologist 

The  three  OCPO  representatives  had  primary  experience  In  the  position 
classification  specialty. 

2.  Ground  rules.  In  general,  the  following  ground  rules  were  agreed  upon: 

a.  Occupational  series  (both  GS  and  WG)  would  be  condensed  Into  20  to  30 

groupings.  Existing  job  families,  as  described  by  the  Office  of  Personnel 
Management  (OPM),  will  not  be  used  as  they  are  too  numerous  (21  GS,  53 
blue-collar)  and  do  not  always  look  at  task  analysis  In  making  occupational 
distinctions.  For  example,  the  GS-6XX  family  Includes  physicians,  nurses, 
medical  machine  technicians,  nursing  assistants,  medical  clerks,  and  other 

positions  which  are  found  In  a  hospital/medical  setting  and  does  not  group 
only  those  positions  with  similar  duties.  This  grouping  Is  made  on  an 
environmental  basis  rather  than  placing  like  tasks  In  the  same  job  family. 

b.  For  the  first  round  of  groupings,  GS-301  would  not  be  considered 

because  of  Its  heterogeneity. 

c.  The  decision  to  put  various  grade  levels  of  one  occupational  series 

Into  more  than  one  grouping  would  be  delayed  until  some  of  the  basic 

determinations  were  made. 

d.  The  first  step  would  be  to  divide  the  job  families  (as  defined  In  OPM 
classification  standards)  Into  one  of  five  large  groups:  professional, 
administrative,  technical,  clerical,  and  blue  collar.  After  this,  future 
breaks  of  those  groups  (by  occupational  series)  would  be  made. 

e.  Originally,  the  group  felt  that  blue-collar  occupations  could  be 

Integrated  with  GS  positions.  However,  It  was  decided  that  blue-collar 

occupations  would  be  considered  separately  as  they  have  predominantly  trades 
and  crafts  tasks. 


3,  Consistency  of  specified  occupational  groupings.  For  cross-validation  of 
the  results,  two  subgroups  were  formed.  Each  subgroup  was  made  up  of  two 
personnel  specialists  and  a  psychologist.  The  subgroups  worked  independently; 
plans  were  made  that  the  two  subgroups  would  reconvene  and  compare 
occupational  groupings.  The  areas  of  disagreement  would  be  resolved  at  that 
time.  The  Subgroup  composition  was  as  follows,  group  leader  listed  first: 


Group  A 
Ms  Carpenter 
Mr  Sheppard 
Mr  Wllbourn 


Group  B 
Mr  Hall 
Capt  Cummings 
Mr  Utterback 


The  two  subgroups  met  separately  on  several  occasions  and  divided  occupational 
series  Into  groups.  There  was  no  discussion  between  the  subgroups  on  progress 
or  problems  encountered.  Both  subgroups  used  an  Air  Force  Occupational 
Inventory  dated  5  February  1979,  the  US  Civil  Service  Commission  Position 
Classification  Standards,  the  USCSC  Handbook  of  Blue  Collar  Occupational 
Families  and  Series,  and  the  X-118  Qualification  Standards  as  resource 
documents. 

4.  General  comments  on  occupational  grouping  process.  On  26  February,  the 
two  subgroups  met  to  compare  findings.  The  two  groups*  work  was  very  similar; 
the  only  significant  difference  being  In  the  separation  of  the  blue-collar 
occupations.  General  comments  on  the  groupings  are  as  follows: 

a.  Only  occupations  that  have  positive  education  requirements  (as 
prescribed  by  the  X-118  Qualification  Standards)  were  placed  In  'professional* 
groupings. 

b.  Uhere  grades  of  the  same  occupational  series,  such  as  In  GS-301, 
GS-501,  etc.,  were  placed  In  different  groups,  the  distinctions  were  made 
between  the  GS-5  and  GS-6  levels  and  between  the  GS-9  and  GS-10  levels.  The 
reasoning  here  was  that  most  classification  standards  draw  distinctions 
between  clerical  and  technical  work  at  the  GS-5/6  stage;  also,  the  GS-9  level 
usually  represents  either  the  advanced  technical  grade  or  the  advanced  trainee 
stage. 

c.  One  group  had  divided  the  blue-collar  occupations  by  environment,  such 
as  fabric  working  and  aircraft  working,  whereas  the  other  group  had  placed 
like  tasks  together.  The  second  group's  suggestions  were  adopted  as  they  were 
more  logical  and  corresponded  to  the  rationale  used  In  dividing  the  GS 
occupations.  For  example,  all  Wage  Supervisory  (WS)  positions  were  In  one 
group,  rather  than  being  Integrated  Into  their  respective  occupational 
series.  This  grouping  was  done  because  the  tasks  on  which  the  WS  employee 
would  be  evaluated  were  not  technical,  but  centered  on  supervisory  duties. 

d.  No  definitions  were  specifically  given  to  the  terms  'technical*  and 
•administrative'  as  far  as  determining  the  occupations  which  were  grouped 
together.  For  example.  It  was  Irrelevant  whether  manpower/personnel  jobs  or 
supply/transportatlon  occupations  were  called  'technical'  or 
•administrative.'  What  was  relevant,  however,  was  that  these  like  occupations 
were  grouped  appropriately. 

e.  Certain  general  occupations,  such  as  GS-301,  GS-501,  etc.,  would  be 
placed  In  different  groupings,  as  appropriate.  These  types  of  occupations 
encompass  such  a  variety  of  occupations  that  It  was  more  appropriate  to  group 
them  with  similar  skills  than  strictly  by  occupational  series.  For  example, 
the  GS-301  series  can  range  from  a  trainee  clerical  position  classified  as 
GS-301-1  to  an  Air  Staff  Manager,  GS-301-18.  Therefore,  using  grade  level 
distinctions  made  In  other  occupational  series  as  guides,  grade  level 
distinctions  In  these  occupations  were  made  at  the  GS-5/6  and  GS-9/10  levels 
(see  paragraph  4b). 
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f.  In  each  of  the  four  major  GS  categories  (professional,  administrative, 
technical,  and  clerical),  there  were  a  number  of  occupations  that  did  not  fall 
Into  any  particular  grouping  and  were  too  small  In  population  to  make  Into  an 
Individual  subgroup  for  that  specific  occupational  series.  They  became  the 
Miscellaneous  Professional,  Administrative  Support,  Miscellaneous  Technical 
and  General  Clerical  job  families. 

5.  Specific  comments  on  final  occupational  groups.  Specific  comments  on  the 
groupings  are  presented  below.  The  23  job  family  categories  and  the 
dccupatlonal  series  Included  within  each  job  family  are  contained  In  Tables  A1 
through  A24. 

a.  Accounting  and  Finance  Technical:  Occupations  support  the  Comptroller 
function  and  are  nonmanagerlal  and  nonprofessional. 

b.  Administrative  Support:  Occupations  support  primarily  technical  and 
administrative  and  some  professional  occupations.  Except  for  very  low  grade 
levels,  all  occupations  in  this  grouping  must  have  substantive  knowledge  of 
the  subject-matter  area.  The  occupations  share  a  requirement  for  researching 
of  regulations  and  directives  and  processing  work  which  has  Impact  on  monetary 
expenditures  and  Impact  on  personnel.  The  exception  Is  Editorial  Assistant, 
GS-1087,  which  was  placed  In  this  group  because  of  Its  requirement  of  strict 
adherence  to  guidelines. 

c.  Arts  and  Recreation  Technical:  This  grouping  Is  relatively 
self-explanatory.  It  consists  of  recreation-type  occupations,  as  well  as 
those  In  the  Information  field  which  do  not  have  positive  education 
requirements.  For  the  purpose  of  performance  evaluation,  these  occupations 
(a)  have  particular  technical  standards  which  must  be  upheld  and  (b)  require 
flexibility  to  respond  to  specific  needs. 

d.  Biological  and  Medical  Technical:  Occupations  do  not  have  positive 
education  requirements  and  support  biological  or  medical  professional 
occupations. 

e.  Biology  Professional:  Self-explanatory. 

f.  Business  Technical:  These  occupations  perform  business-  and 
Industry-related  functions  In  the  Air  Force.  The  supply/transportation 
occupations  shown  here  were  not  grouped  In  the  Supply/Transportation  Group  as 
the  category  Is  oriented  toward  management  positions.  At  the  GS-6/9  levels, 
the  supply/ transportation  positions  are  technical. 

g.  Computer  Technical:  Self-explanatory. 

h.  Engineering,  Physical  Science,  and  Mathematical  Professional:  The 
similarity  In  tasks  of  dealing  with  physical  phenomena  and  mathematics  drew 
these  occupations  Into  the  same  grouping.  All  occupations  have  positive 
education  requirements. 


i.  Engineering,  Physical  Science,  and  Mathematical  Technical:  Technical 
support  of  groups  listed  In  paragraph  h;  do  not  have  positive  education 
requirements. 

j.  General  Clerical:  These  occupations  are  similar  as  to  what  they  do 
not  require.  For  the  most  part,  they  do  not  require  keyboard  skills, 
substantive  knowledge  of  subject-matter  area,  or  extensive  specialized  or 
specific  experience  In  the  type  of  work  to  be  performed.  The  Intelligence 
Clerk  GS-134,  Printing  Clerk  GS-351,  Communications  Clerk  GS-394,  and  Medical 
Clerk  GS-679  are  In  series  which  denote  their  environments  rather  than 
specialized  tasks. 

k.  General  Management/Admlnlstratlon:  These  occupations  represent  those 
series  In  which  general  management  jobs  are  found.  These  may  be  found  In  any 
organization;  management  skills,  rather  than  specialized  knowledges,  are  their 
predominant  requirements. 

l.  Keyboard  Clerical:  The  Secretary  GS-316  occupation  was  Included  as 
these  positions  usually  require  keyboard  skills  as  well  as  secretarial 
skills.  Additionally,  the  career  pattern  for  secretaries  Is  usually  an 
outgrowth  of  the  GS-312/316  series. 

m.  Medical  Professional:  Self-explanatory. 

n.  Miscellaneous  Professional:  These  occupations  fall  into  the  other 
professional  category.  There  are  obvious  subgroups  here  (GS-1701,  1710,  1720, 
1725;  GS-184,  185,  190,  193);  however,  these  subgroups  have  populations  that 
are  too  small  for  performance  appraisal  purposes  and  therefore  were  not 
divided  further. 

o.  Miscellaneous  Technical:  The  same  rationale  as  above  (paragraph  n) 
applies  for  these  technical  positions,  which  do  not  fall  clearly  In  any  other 
grouping. 


p.  Personnel  and  Manpower:  Self-explanatory. 

q.  Specialized  Management/Administration:  These  occupations  have 

management  orientations  but  In  specific  subject  matter  areas.  Although  these 
specializations  are  not  necessarily  similar,  the  requirement  for  management 
within  certain  subject-matter  requirements  Is  present  In  all. 

r.  Supply/Transportation:  Self-explanatory  (see  paragraph  5f). 

s.  Basic  Shop  Operations:  These  occupations  do  not  require  specialized 
skills. 


t.  Electric  Repair  and  Service:  Self-explanatory. 

u.  General  Equipment  Operation:  Self-explanatory. 
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Table  A-l.  Occupational  Group 


Job  Fanlly 


ENGINEERING,  PHYSICAL  SCIENCE,  AND  MATHEMATICAL  PROFESSIONAL 

MEDICAL  AND  BIOLOGY  PROFESSIONAL 

MISCELLANEOUS  PROFESSIONAL 

COMPUTER  TECHNICAL 

PERSONNEL  AND  MANPOWER 

SUPPLY  AND  TRANSPORTATION 

BIOLOGICAL  AND  MEDICAL  TECHNICAL 

MISCELLANEOUS  TECHNICAL 

ENGINEERING,  PHYSICAL  SCIENCE,  AND  MATHEMATICAL  TECHNICAL 

ACCOUNTING  AND  FINANCE  TECHNICAL 

BUSINESS  TECHNICAL 

ARTS  AND  RECREATION  TECHNICAL 

GENERAL  MANAGEMENT/ ADMINISTRATION 

SPECIALIZED  MANAGEMENT/ADMINISTRATION 

ADMINISTRATIVE  SUPPORT 

GENERAL  CLERICAL 

KEYBOARD  CLERICAL 

WAGE  SUPERVISORY 

ELECTRIC  REPAIR  AND  SERVICE 

GENERAL  MECHANICAL 

BASIC  SHOP  OPERATIONS 

SYSTEMS  REPAIR  AND  SERVICES 

GENERAL  EQUIPMENT  OPERATIONS 


Table  A-2.  Job  Family  1 


ENGINEERING,  PHYSICAL  SCIENCE,  AND  MATHEMATICAL  PROFESSIONAL 

Occupational 

Series  Title 


405  Pharmacology 

408  Ecology 

410  Zoology 

434  Plant  Pathology 

435  Plant  Physiology 

437  Textile  Technology 

801  General  Engineering 

803  Safety  Engineer 

804  Fire  Prevention  Engineer 

806  Materials  Engineer 

807  Landscape  Architect 

808  Architect 

810  Civil  Engineer 

819  Environmental  Engineer 

830  Mechanical  Engineer 

840  Nuclear  Engineer 

850  Electrical  Engineer 

855  Electronic  Engineer 

858  Biomedical  Engineer 

861  Aerospace  Engineer 

881  Petroleum  Engineer 

892  Ceramic  Engineer 

893  Chemical  Engineer 

894  Welding  Engineer 

895  Industrial  Engineer 

1301  General  Physical  Science 

1306  Health  Physics 

1310  Physics 

1313  Geophysics 

1320  Chemistry 

1321  Metallurgy 

1330  Astronomy/ Space  Science 

1340  Meteorology 

1370  Cartographer 

1372  Geodesist 

1373  Land  Surveyer 

1384  Textile  Technology 

1386  Photographic  Technology 

1510  Actuary 

1515  Operations  Research 

1520  Mathematics 

1529  Mathematical  Statistics 

1530  Statistician 

1540  Cryptography 

1550  Computer  Science 


Table  A-3.  Job  Family  2 


MEDICAL  AND  BIOLOGY  PROFESSIONAL 


Occupational 

Series  Title 


401  Biology 

403  Microbiology 

413  Physiology 

414  Entomology 

430  Botany 

440  Genetics 

454  Range  Conservation 

457  Soil  Conservation 

460  Forestry 

471  Agronomy 

486  Wildlife  Biology 

601  General  Health  Science 

602  Medical  Officer 

603  Physician's  Assistant 

610  Nurse 

630  Dietitian  and  Nutritionist 

631  Occupational  Therapy 

633  Physical  Therapist 

635  Corrective  Therapy 

637  Manual  Acts  Therapy 

638  Recreation/Creative  Arts  Therapist 

639  Educational  Therapy 

644  Medical  Technologist 

660  Pharmacist 

662  Optometrist 

665  Speed  Pathology/Audlology 

668  Podiatrist 

680  Dental  Officer 

690  Industrial  Hygienist 

701  Veterinarian 


Table  A-4.  Job  Family  3 


MISCELLANEOUS  PROFESSIONAL 


Occupational 

Series 

Title 

020 

Community  Planner 

060 

Chaplain 

101 

Social  Science 

110 

Economist 

131 

International  Relations 

140 

Manpower  Research 

150 

Geography 

160 

Civil  Rights  Analysis 

170 

Historian 

180 

Psychologist 

184 

Sociologist 

185 

Social  Worker 

190 

Anthropologist 

193 

Archeology 

493 

Home  Economist 

510 

Accountant 

9C5 

Attorney 

1015 

Museum  Curator 

1221 

Patent  Advisor 

1222 

Patent  Attorney 

1410 

Librarian 

1420 

Archivist 

1701 

General  Education/Training 

1710 

Educatlon/Vocatlonal/Tralnlng 

1720 

Education  Research  and  Program 

1725 

Public  Health  Education 

Table  A-5.  Job  Family  4 

COMPUTER  TECHNICAL 

Occupational 

Series 

Title 

332 

334 

335 


Computer  Operator 
Computer  Specialist 
Computer  Clerk/Assistant 


Occ 

upa 

tlonal 

Ser 

1e$ 

Table  A-6.  Job  Family  5 


PERSONNEL  AND  MANPOWER 


Manpower  Development 
Personnel  Management 
Military  Personnel  Management 
Personnel  Staffing 
Position  Classification 
Occupational  Analysis 
Salary  and  Wage  Administration 
Employee  Relations 
Labor  Relations 
Employee  Development 
Contractor  Industrial  Relations 
Equal  Employment  Opportunity 


Table  A-7.  Job  Family  6 


SUPPLY  AND  TRANSPORTATION 


Occupational 


Transportation  Industry  Analyst 
General  Supply  Grades  10  and  above 
Supply  Program  Management 
Supply  Technician  Grades  10  and  above 
Inventory  Management 

Distribution  Facllltles/Storage  Management 

Supply  Cataloging 

Transportation 

Traffic  Management 

Transportation  Operations 


Table  A-8.  Job  Faally  7 


Occupational 


BIOLOGICAL  AND  MEDICAL  TECHNICAL 


Biological  Technician 

Forestry  Technician 

Practical  Nurse 

Nursing  Assistant 

Medical  Supply  Aid/Technician 

Autopsy  Assistant 

Nuclear  Medicine  Technician 

Medical  Technician 

Pathology  Technician 

Diagnostic  Radiologic  Technologist 

Therapeutic  Radiologic  Technologist 

Medical  Machine  Technician 

Pharnacy  Technician 

Restoration  Technician 

Orthotlst/Prosthetlst 

Medical  Record  Librarian 

Medical  Record  Technician 

Dental  Assistant 

Dental  Hygienist 

Dental  Lab  Assistant/Technician 

Sanitarian 

Envlroniaental  Health  Technician 
Health  Aid/Technician 


Table  A-9.  Job  Family  8 


MISCELLANEOUS  TECHNICIAN 


Occupational 

Series 

Title 

021 

Community  Planning  Technician 

028 

Environmental  Protection  Specialist 

050 

Funeral  Directing 

062 

Clothing  Design 

081 

Fire  Protection 

083 

Police 

085 

Guard 

102 

Social  Service  Technician 

119 

Economics  Assistant 

132 

Intelligence 

181 

Psychology  Technician 

186 

Social  Services  Assistant 

301 

Administrative  Series  Grades  6-9 

392 

General  Communication 

393 

Communications  Specialist 

671 

Health  System  Specialist 

962 

Contract  Representative 

1021 

Office  Drafting 

1361 

Navigational  Information 

1411 

Library  Technician 

1412 

Technical  Information  Services 

1421 

Archives  Technician 

1721 

Training  Instructor 

1810 

Investigating 

1811 

Criminal  Investigating 

1815 

Air  Safety 

1825 

Aviation  Safety 

1910 

Quality  Assurance 

1960 

Quality  Inspection 

2032 

Packaging 

2144 

Cargo  Scheduler 

2152 

Air  Traffic  Controller 

2181 

Aircraft  Operating 
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Table  A-12.  Job  Faiilly  11 


BUSINESS  TECHNICAL 


Occupational 

Series 

Title 

809 

Construction  Control 

1101 

General  Business  and  Industry  Grades  6-9 

1302 

Contract/Procurement 

1104 

Property  Disposal 

1105 

Purchasing 

1106 

Procurement  Clerk/Assistant 

1107 

Property  Disposal  Clerk/Technician 

1130 

Public  Utilities  Specialist 

1150 

Industrial  Specialist 

1152 

Production  Controller 

1170 

Realty 

1670 

Equipment  Specialist 

2001 

General  Supply  Grades  6-9 

2005 

Supply  Clerk/Technician  Grades  6-9 

2102 

Transportation  Clerk/Assistant 

2131 

Freight  Rate  Clerk 

2132 

Travel  Clerk 

Table  A-13.  Job  Family  12 

ARTS  AND  RECREATION  TECHNICAL 

Occupational 

Series 

Title 

030 

Sports  Specialist 

188 

Recreation  Specialist 

189 

Recreation  Aid/Technician 

1001 

General  Arts  and  Information 

1010 

Exhibits  Specialist 

1016 

Museum  Special ist/Technlclan 

1020 

Illustrating 

1035 

Public  Affairs  (GS-1081s  will  exist  until  Apr  82) 

1046 

Language  Specialist 

1056 

Art  Specialist 

1060 

Photographer 

1071 

A-Y  Production  Specialist 

1082 

Writer-Editor 

1083 

Technical  Writer/Editor 

1084 

Visual  Information 

Table  A-14.  Job  Family  13 


GENERAL  MANAGEMENT/AOMINISTRATION 


Occupational 

Series 

Title 

301 

Miscellaneous  Administration  and  Program  Grades  10-18 

340 

Program  Manager 

341 

Administrative  Officer 

342 

Support  Services  Administrator 

343 

Management  Analyst 

345 

Program  Analyst 

Table  A-15.  Job  Family  14 

SPECIALIZED  MANAGEMENT/ADMINISTRATION 

Occupational 

Series 

Title 

018 

Safety  and  Occupational  Health  Management 

080 

Security  Administration 

346 

Logistics  Manageient 

391 

Comunlcatlons  Management 

501 

Accounting  Grades  10-18 

505 

Financial  Management 

670 

Health  System  Administration 

673 

Hospital  Housekeeping  Management 

1101 

General  Business  and  Industry  Grades  10-18 

1103 

Industrial  Property  Management 

1140 

Facility  Management 

1144 

Commissary  Management 

1173 

Housing  Management 

1176 

Building  Management 

1601 

General  Facilities  and  Equipment 

1640 

Facilities  Management 

1654 

Printing  Management 

1658 

Laundry/Dry  Cleaning  Plant  Management 

1666 

General  Housekeeping 

1667 

Steward 
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Table  A-16.  Job  Family  15 


ADMINISTRATIVE  SUPPORT 


Occupational 


Series 


Title 


019  Safety  Technician 

029  Environmental  Protection  Assistant 

187  Social  Services 

203  Personnel  Clerk/Assistant 

204  Military  Personnel  Clerk/Technlclan 

344  Management  Clerk/Technlclan 

950  Paralegal  Specialist 

963  Legal  Instrument  Examiner 

986  Legal  Clerk/Technlclan 

990  Claims  Examiner 

992  Loss  and  Damage  Claims  Examiner 

995  Dependents  and  Estate  Claims  Examiner 

998  Claims  Clerk 

1087  Editorial  Assistant 

1101  General  Business  and  Industry  Grades  1-5 

1702  Education/Training  Technician 

2135  Transportation  Loss/Damage  Claims  Examiner 


Table  A-17.  Job  Family  16 


GENERAL  CLERICAL 


Occupational 

Series 

Title 

134 

Intelligence  Clerk 

302 

Messenger 

303 

Miscellaneous  Clerk/Assistant 

304 

Information  Receptionist 

305 

Mall  and  File 

309 

Correspondence 

350 

Office  Machine  Operator 

351 

Printing  Clerk 

357 

Coding 

394 

Communications  Clerk 

501 

General  Accounting  Grades  1-5 

679 

Medical  Clerk 

2001 

General  Supply  Grades  1-5 

2005 

Supply  Clerk/Technlclan  Grades  1-5 

2091 

Sales  Store  Clerk 

2134 

Shipment  Clerk 

2151 

Dispatcher 
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Table  A-18.  Job  Fa*11y  17 


KEYBOARD  CLERICAL 


Occupational 

Series 

Title 

312 

Clerk-Stenographer 

313 

Work  Unit  Supervisor 

318 

Secretary 

319 

Closed  Microphone  Reporter 

322 

Clerk-Typist 

354 

Bookkeeping  Machine  Operator 

355 

Calculating  Machine  Operator 

356 

Data  Transcriber 

359 

EAtf  Operator 

382 

Telephone  Operator 

385 

Teletypist 

388 

Crypotgraphlc  Equipment  Operator 

389 

Radio  Operating 

390 

Comunlcatlons  Relay  Operator 

1046 

Language  Clerk 

Table  A-19.  Job  Fanlly  18 

WAGE  SUPERVISORY  (WS) 

Occupational 

Series 


Title 


Table  A-20.  Job  Family  19 


ELECTRIC  REPAIR  AND  SERVICE 


Occupational 

Series 

Title 

2SOO 

Wire  Communications  Equipment  Installation  and  Maintenance 

2600 

Electronic  Equipment  Installation  and  Maintenance 

2800 

Electrical  Installation  and  Maintenance 

Table  A-21.  Job  Family  20 

GENERAL  MECHANICAL 

Occupational 

Series 

Title 

3100 

Fabric  and  Leather  Work 

3400 

Machine  Tool  Work 

3600 

Structural  and  Finishing  Work 

3700 

Metal  Processing 

3800 

Metal  Work 

3900 

Motion  Picture,  Radio,  Television,  and  Sound  Equipment 

4000 

Lens  and  Crystal  Work 

4100 

Painting  and  Paperhanging 

4200 

Plumbing  and  Pipefitting 

4300 

Pliable  Materials  Work 

4600 

Woodwork 

4800 

General  Equipment  Maintenance 

6500 

Ammunition,  Explosive,  and  Toxic  Material  Work 

8800 

Aircraft  Overhaul 

9000 

Film  Processing 

Table  A-22.  Job  Faiilly  21 


BASIC  SHOP  OPERATIONS 


Occupational 

Series 

Title 

3500 

General  Services  and  Support  Work 

4400 

Printing 

5000 

Plant  and  An1»al  Work 

5200 

Miscellaneous  Occupations 

6900 

Warehousing  and  Stockhandling 

7000 

Packing  and  Processing 

7300 

Laundry,  Dry  Cleaning,  and  Pressing 

7400 

Food  Preparation  and  Serving 

7600 

Personal  Services 

Table  A-23.  Job  Faulty  22 

SYSTEMS  REPAIR  AND  SERVICES 

Occupational 

Series 

Title 

3300 

Instrunent  Work 

5300 

Industrial  Equlpiient  Maintenance 

5800 

Transportation/Hoblle  Equlpiient  Maintenance 

6600 

Amaiient  Work 

8200 

Fluid  Systems  Maintenance 

8600 

Engine  Overhaul 

Table  A-24.  Job  Family  23 

GENERAL  EQUIPMENT  OPERATION 

Occupational 

Series 

Title 

4700 

General  Maintenance  and  Operations 

5400 

Industrial  Equlpiient  Operation 

5700 

Transportation/Hoblle  Equipment  Operation 
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APPENDIX  B:  DEVELOPMENT  AND  ADMINISTRATION  OF  THE 
CIVILIAN  POTENTIAL  APPRAISAL  SYSTEM 
DATA  COLLECTION  INSTRUMENTS 

I.  OVERVIEW 

This  appendix  describes  the  Air  Force's  stated  promotion  system 
requirements,  legal  considerations,  and  development  and  field  testing  of  the 
data  collection  materials  for  development  of  the  Civilian  Potential  Appraisal 
System  (CPAS).  The  CPAS  development  process  used  a  "funnel Ing*  type  approach, 
whereby  an  extremely  large  number  of  promotion  factors  were  considered  In  the 
initial  pool  of  candidate  factors.  This  Initial  set  of  factors  was  reduced  at 
subsequent  decision  points,  under  a  variety  of  different  considerations.  Data 
were  obtained  on  the  Initial  candidate  factors  by  administering  aptitude 
measures,  self-report  questionnaires,  and  peer  and  supervisory  ratings. 
Development  of  the  data  collection  instruments,  the  Civilian  Personnel 
Examination  (CPE),  Score -Check  Ing  Test  (SCT),  Demographics 

Questionnaire-Worker,  and  Job/Worker  Characteristics  Ratings  booklet.  Is 
described.  After  pretesting,  the  Demographics  Questionnaire-Worker  was 
administered  to  a  CONUS-wlde  sample  of  20,000  workers,  and  the  Job/Worker 
Characteristics  Ratings  booklets  were  administered  both  to  the  target  workers' 
supervisors  and  20,000  co-workers.  Responses  to  the  data  collection  effort 
yielded  12,865,  or  64.32X,  complete  and  usable  data  sets.  These  data  sets 
subsequently  were  used  In  analyses  to  Identify  the  final  set  of 
job-family-specific  promotion  ranking  factors  and  their  relative  weights  as 
described  In  Appendix  C.  A  subsample  of  2,000  employees  were  administered  the 
CPE  and  SCT  for  research  purposes  only. 

II.  DESIGN  CONCEPT 
Need  for  the  Civilian  Potential  Appraisal  System 

By  1976,  the  Air  Force  Directorate  of  Civilian  Personnel  (USAF/MPK)  had 
become  acutely  aware  of  Inherent  problems  In  the  current  civilian  appraisal 
program.  The  existing  appraisal  system's  Inflated  ratings  had  Insufficient 
variance  to  make  distinctions  among  employees  and  as  a  result,  the  systems 
were  not  viewed  as  being  credible  by  the  employees,  supervisors,  or 
management.  In  a  December  1976  request  for  personnel  research,  USAF/MPK 
pointed  out  the  need  for  separate  appraisals  -  those  to  be  used  for  promotion 
purposes  versus  those  used  for  other  personnel  actions. 

Although  appraisal  Instruments  employed  In  current  and  past  operational 
programs  were  developed  to  serve  a  variety  of  purposes  (e.g.,  selection  for 
training,  details,  reassignments,  merit  promotions),  their  use  for  the  latter 
purpose  --  merit  promotion  --  Inevitably  led  to  their  Ineffectiveness  In 
achieving  any  of  those  purposes.  Of  critical  Importance  was  the  ultimate  loss 
of  credibility  In  the  appraisal  program  In  the  eyes  of  the  employees, 
supervisors,  and  higher  management  (RPR  76-40,  pp.  1-2). 

This  concern  led  AFHRL  to  develop  separate  promotion  appraisal  and  job 
performance  appraisal  systems  for  the  assessment  of  Air  Force  civilian 
employees.  One  set  of  appraisals  (t.e. ,  CPAS)  was  developed  strictly  for 
promotion  purposes,  while  a  second  set  (l.e.,  SEAS,  GMAS,  JPAS)  was  developed 
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to  assess  performance  In  relation  to  specified  job  performance  standards.  The 
job  performance  ratings  were  to  be  used  for  other  personnel  actions  such  as 
,  within  grade  Increases,  feedback,  dismissal,  retraining,  merit  pay,  and  award 
of  bonuses  (Cowan,  Thompson,  &  Guerrieri,  1983;  Guerrieri,  1981;  Guerrierl, 
Thompson,  Cowan,  &  Gould,  1984). 

Initial  Design  of  the  System 


Three  basic  sets  of  requirements  guided  the  development  of  CPAS  —  those 
of  the  user  (USAF/MPK),  the  U.S.  Civil  Service  Commission,  and  legal 
guidelines.  USAF/MPK  wanted  a  system,  separate  from  the  job  performance 
appraisal  systems,  that  measured  employees'  potential  to  perform  at  levels 
higher  than  the  requirements  of  their  current  jobs.  The  desired  promotion 
system  was  to  be  Inflation-resistant,  as  simple  as  possible,  and  compatible 
for  use  in  an  automatic  data  processing  environment.  The  U. S.  Civil  Service 
Commission  (now  the  Office  of  Personnel  Itanagement)  regulations  precluded  "(1) 
Withholding  information  from  employees  on  appraisal  ratings  or  any  other 
Information  concerning  their  job  performance;  (2)  Using  a  'single*  total  score 
for  each  employee  for  all  promotion  considerations  (i.e.,  the  score  used  in 
each  consideration  must  be  composed  of  those  elements  in  an  appraisal  form 
that  are  relevant  to  the  position  to  be  filled);  and  (3)  Using  elements 
Inherently  descriptive  of  the  'personality'  of  an  employee  rather  than  as 
attributes  relevant  to  job  performance  *  (RPR  ;6-40,  p.2). 

Military  selection  and  promotion  systems  have  been  exempted  from  the 
jurisdiction  of  civil  law  by  the  Office  of  Management  and  Budget,  but  similar 
systems  for  Department  of  Defense  civilians  have  no  such  exemptions.  The  mood 
of  the  Nation  for  equal  employement  opportunity  and  the  growing  number  of 
legal  challenges  to  civilian  promotion  systems  Indicated  a  growing  probability 
that  any  promotion  system  would  eventually  be  challenged  In  court.  Therefore, 
the  CPAS  developers  made  an  Initial  assumption  that  the  system  they  developed 
would  be  challenged  In  court.  A  top  priority,  then,  was  to  develop  a  legally 
defensible  system  which  met  the  requirements  under  the  Uniform  Guidelines  for 
Employee  Selection  of  1978. 

An  initial  review  of  relevant  court  cases  (United  States  Office  of 
Personnel  Management,  1979)  and  discussions  with  expert  Industrial 
psychologists  (Mullins,  1981;  Mullins  &  Winn,  1979)  concluded  that  ratings 
used  for  promotion  purposes  had  to  be  as  objective  and  job  related  as 
possible.  Asking  supervisors  to  rate  directly  employees'  potential  to  perform 
higher  level  jobs,  when  they  were  not  familiar  with  those  jobs  and  their 
requirements,  would  probably  not  be  legally  defensible.  Further,  there  was 
little  empirical  evidence  that  supervisors  are  able  to  make  projections  of 
future  performance.  It  was  believed  that  job-relevant  aptitude  scores  and 
verifiable  experience,  training,  and  other  demographic  information  would  be 
the  most  objective  predictors  of  future  performance.  In  addition,  it  was 
thought  that  a  combination  of  peer,  supervisor,  and  alternate  supervisor 
ratings  of  current  job  performance  should  also  be  used  to  complete  the 
prediction  system. 
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The  Uniform  Guidelines  for  Employee  Selection  of  1978  became  available  In 
1979,  and  early  court  cases  began  to  define  operationally  the  meaning  of  the 
provisions  (Casclo  &  Bernardln,  1981).  In  effect,  the  Guidelines  say  that  a 
selection  or  promotion  system  that  creates  adverse  Impact  against  a  minority 
or  protected  group  of  employees  or  potential  employees  Is  legally  defensible 
If  the  system's  development  process  considered  all  practical  variables  which 
are  likely  to  be  related  to  job  performance  and  selected  those  which  are  most 
valid  (content,  construct,  or  predictive  validity)  but  are  least  likely  to 
create  adverse  Impact.  Hence,  the  design  concept  for  developing  CPAS  was  to 
obtain  the  maximum  Information  on  each  employee  from  a  variety  of  different 
sources  and  Identify  the  smallest  subset  which  minimized  adverse  Impact  while 
providing  a  valid  system.  To  meet  this  objective,  a  system  of  three  diverse 
appraisal  sources  was  developed  —  testing,  background  Information,  and 
ratings. 

1.  Aptitude  Measures.  To  obtain  relevant  test  data,  two  tests  were 
developed  for  this  specific  use,  the  Civilian  Personnel  Examination  (CPE)  and 
the  Score-Checking  Test  (SCT).  The  CPE  test  would  provide  both  specific  and 
general  aptitude  data  applicable  to  a  variety  of  jobs,  while  the  SCT  would 
measure  the  ability  to  pay  attention  to  detail  for  record-keeping/reviewing 
types  of  clerical  jobs.  The  CPE  .  was  a  modification  of  the  Airman 
Qualification  Examination,  Form  J  (AQE-J)  (V 1  tola,  Massey,  &  Wllbourn,  1971) 
that  was  updated  and  adapted  for  civilian  personnel.  The  examination 
consisted  of  11  subtests,  designed  to  test  the  worker's  knowledge/aptitude  In 
a  variety  of  areas  (see  Table  B-l).  The  subtests  most  heavily  emphasized 
blue-collar  and  clerical  skills,  although  many  of  the  subtests  (e.g.,  reading 
ability,  arithmetic  reasoning,  word  knowledge)  could  apply  to  a  variety  of  job 
series.  The  Score-Checking  Test  (SCT)  was  a  carefulness  test  (Mullins  » 
Force,  1962).  This  test  required  the  examinee  to  check  two  sets  of  numbers 
for  transcribing  errors  and  was  designed  to  evaluate  that  person's  carefulness 
In  spotting  these  discrepancies. 

Table  B-l.  Civilian  Personnel  Examination  (CPE)  Subtests 


Arithmetic  Computation  consists  of  simple  arithmetic  Items  Involving 
addition,  subtraction,  multiplication,  and  division  of  whole  numbers. 
Administered  as  a  speed  test,  It  Is  designed  to  measure  the  ability  to 
manipulate  numbers  rapidly  and  accurately.  (60  Items,  8  minutes) 

Arithmetic  Reasoning  evaluates  the  examinee's  ability  to  think  through 
mathematical  problems  presented  In  verbal  form.  It  Involves  the  discovery  and 
application  of  the  general  mathematical  principles  required  to  arrive  at  a 
correct  solution  to  each  problem,  as  well  as  performance  of  the  necessry 
calculations  to  attain  the  solution.  (15  Items,  25  minutes) 

Data  Interpretation  Is  designed  to  measure  the  ability  to  draw  conclusions 
or  make  Interpretations  from  data  presented  in  the  form  of  graphs,  charts,  and 
tables.  Two  or  three  Items  are  based  on  each  presentation.  (10  items,  13 
minutes) 
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Table  B-1  (Continued) 


Electrical  Information  Involves  the  ability  to  apply  previously  acquired 
knowledge  In  the  areas  of  electricity  and  electronics  toward  the  solution  of 
problems  In  practical  situations.  (15  Items,  9  minutes) 

Mechanical  Information  consists  of  verbal  Items  relating  to  the 
understanding  and  application  of  basic  techniques  required  for  troubleshooting 
and  repairing  various  mechanical  devices.  It  also  requires  the  examinee  to 
determine  operating  characteristics  from  pictures  of  mechanical  devices.  (15 
Items,  12  minutes) 

Hidden  Figures  requires  the  examinee  to  determine  which  one  of  five  simple 
line  drawings  Is  contained  In  a  more  complex  arrangement  of  geometric 
figures.  These  Items  appear  within  the  test  In  sets  of  four  Items  per  page 
with  the  five  simple  line  drawings  at  the  top  of  the  page.  (15  Items,  20 
minutes) 

Pattern  Comprehension  Involves  visualizing  the  folding  of  flat  patterns 
Into  three-dimensional  objects  and  subsequently  determining  the  location  of 
specific  points  which  are  common  to  both  the  pattern  and  the  solid  figure. 
These  Items  appear  within  the  test  In  groups  of  three  items  for  each  pattern. 
(18  Items,  12  minutes) 

Shop  Practices  Is  a  pictorial  test  which  requires  the  examinee  to  Identify 
pictured  tools  and  determine  their  proper  use  In  a  specific  situation  or  the 
selection  of  the  proper  tool  for  use  In  a  given  task.  (15  Items,  9  minutes) 

Word  Knowledge  Is  a  test  of  verbal  ability  Involving  the  definition  of 
words:  this  Is  a  classical  vocabulary  test  of  non-technical  terms.  (30 
Items,  10  minutes) 

Verbal  Analogies  Is  designed  to  measure  the  ability  to  perceive 
relationships.  It  requires  the  examinee  to  complete  a  relationship  of  four 
terms  so  that  the  correspondence  between  the  last  two  terms  will  be  the  same 
as  that  between  the  first  two  (given)  terms;  i.e.,  dark  Is  to  light  as  night 
Is  to  .  (20  Items,  10  minutes) 


Reading  Comprehension  evaluates  the  examinee's  ability  to  read  and 
understand  a  written  passage.  It  requires  the  examinee  to  read  a  series  of 
written  passages  and  then  answer  a  series  of  questions  on  each  passage.  (18 
Items,  25  minutes) 


The  use  of  testing  Information  provides  three  particular  benefits  In  a 
personnel  appraisal  system.  First,  tests  are  widely  used  and  accepted  by 
personnel  psychologists  because  of  their  high  validities  In  predicting 
successful  outcomes  In  a  variety  of  situations  (Green,  1978).  Second,  tests 


provide  a  different  sort  of  information  than  that  normally  available  to  the 
supervisor  and  other  raters  (i.e. ,  more  comprehensive  Information  on  a  larger 
number  of  the  employee's  aptitudes  and  abilities,  other  than  those  being  used 
in  the  current  job).  Third  (and  perhaps  most  Important  In  the  present 
context),  tests  are  relatively  inflation-free.  Therefore,  It  was  decided  that 
testing  data  would  provide  a  particularly  useful  Input  to  the  appraisal 
process. 

2.  Demographic/Background  Information.  The  second  source  of  relevant  data 
to  be  used  in  CPAS  was  provided  by  the  Demographic  Questionnaire-Worker.  The 
Demographic  Questionnaire-Worker,  which  was  designed  to  be  completed  by  the 
employee,  contained  102  items  on  education  level,  sex  and  ethnic  group,  time 
in  civil  service  and  grade,  awards  history,  desire  for  a  promotion,  and 
related  areas.  The  questionnaire  also  contained  a  listing  of  potentially 
relevant  high  school/college/trade  school  courses  that  could  be  used  In 
promotion  selections.  A  copy  of  the  Demographic  Questionnaire-Worker  Is 
contained  In  Appendix  B-l. 

This  questionnaire  was  believed  to  be  the  most  efficient  method  for 
obtaining  basic  data  Important  for  a  valid  promotion  system.  A  long  history 
of  studies  with  demographic  and  other  self -report-type  questionnaires  has 
demonstrated  their  validity  and  usefulness  In  selection  or  promotion  systems 
(Reilly  &  Chao,  1980).  Reilly  and  Chao  demonstrated  that  self-report 
background  measures  provide  as  much.  If  not  more,  validity  than  do  many 
alternative  selection  devices,  with  minimal  adverse  Impact.  Therefore, 
demographic  data  seemed  to  provide  a  valuable  component  for  an  appraisal 
system.  Many  of  the  variables  Included  In  the  questionnaire,  such  as  race  and 
sex,  had  no  potential  for  actual  use  in  CPAS  but  were  Included  to  aid  analysis 
of  the  obtained  data. 

3.  Ratings.  The  third  source  of  Information  to  be  obtained  and  used  In 
CPAS  was  to  be  provided  by  the  Job/Worker  Characteristics  Ratings  booklet. 
Since  this  booklet  played  a  key  role  In  the  development  of  the  final 
operational  CPAS,  the  design  of  the  booklet  will  be  discussed  In  more  detail 
In  the  following  paragraphs.  This  booklet  Is  reprinted  In  Appendix  B-2. 

III.  DESIGN  OF  THE  JOB/WORKER  CHARACTERISTICS  RATINGS  BOOKLET 

A  wide  variety  of  Items  were  developed  and  placed  Into  10  sections  of  the 
Job/Worker  Characteristics  Ratings  booklet.  This  booklet  was  specifically 
designed  to  be  completed  by  several  different  raters  who  were  familiar  with 
the  employee's  job  performance.  Each  of  the  10  sections  listed  In  Table  B-2 
tapped  a  different  type  of  Information  (e.g.,  overall  job  performance,  ratings 
on  trait  factors,  ratings  on  general  abilities,  and  rater  background 
information).  The  primary  objective  of  the  ratings  portion  of  the  CPAS  data 
collection  Instruments  was  to  provide  three  types  of  data  necessary  for  the 
development  of  an  objective,  reliable,  and  valid  appraisal  process:  normative 
data  on  the  worker  and  that  person's  job  performance,  Ipsatlve  rating  data  on 
the  worker,  and  rater  accuracy  data. 


Table  B-2.  Job/Worker  Characteristics  Ratings 


Section 

Contents 

I 

Job  Performance  Data 

II 

Estimated  Score  on  Vocabulary  Test 

III 

Ratings  of  Trait  Factors 

IV 

Ratings  of  General  Abilities 

V 

Rankings  of  Trait  Factors 

VI 

Rankings  of  General  Abilities 

VII 

Additional  Information 

VIII 

Rankings  of  Job  Characteristics  or  Trait  Factors 

X 

Background  Data 

Normative  Data 

Normative  ratings  are  those  that  compare  the  job-incumbent's  performance 
to  some  reference  group  (such  as  other  employees  in  the  same  job  In  the  same 
section),  and  the  ratings  reflect  the  placement  of  that  Individual  In  terms  of 
relative  standing  with  the  reference  group.  In  the  development  of  CPAS,  it 
was  believed  that  normative  ratings  of  each  employee  would  be  useful  in 
determining  promotion  potential.  The  scale  format  to  be  used  In  obtaining 
these  normative  ratings  was  a  question  to  be  addressed.  The  mass  of  research 
on  job  performance  ratings  has  failed  to  demonstrate  that  any  specific  set  or 
type  of  rating  statement  significantly  enhances  prediction  accuracy.  For 
example,  behavlorally  anchored  rating  scales  (BARS)  have  been  extensively 
researched  In  recent  years.  However,  evidence  for  the  psychometric 
superiority  of  these  scales  Is  mixed,  and  their  predictive  power  Is  virtually 
untested  (Dunnette  &  Borman,  1979).  Similarly,  Massey,  Mullins,  and  Earles 
(1978)  failed  to  detect  any  difference  between  worker-oriented,  task-oriented, 
and  trait-oriented  rating  scales  In  predicting  NCO  Academy  performance. 
Curton,  Ratliff,  and  Mullins  (1979)  likewise  found  no  differences  between  sets 
of  rating  factors,  even  though  subject-matter  experts  had  preselected  separate 
sets  of  factors  that  seemed  most  and  least  relevant  to  performance.  Although 
certain  rating  contents  or  techniques  may  ultimately  provide  superior  ratings, 
thesa  scales  have  not  yet  been  developed. 

Since  no  definitive  results  suggested  a  particular  format  or  job-specific 
content  for  appraisal  rating  scales,  a  working  group  of  AFHRL  behavioral 
scientists  selected  a  set  of  general  performance  ratings  and  a  set  of  trait 
factors  that  would  apply  to  all  civilian  job  families  In  the  Air  Force.  It 
was  not  Intended  that  every  factor  would  apply  to  every  job  family  (for  a 
description  of  the  job  families,  see  Appendix  A)  but  rather,  that  the  factors 
would  cover  the  Important  components  of  each  job  family.  These  ratings  and 
factors  were  similar  to,  and  extracts  of,  those  commonly  used  In  research  and 
operational  systems  (e.g.,  Massey,  Mullins,  &  Earles,  1978;  Roach,  1956).  The 
general  performance  ratings  are  presented  in  Section  I  of  the  Job/Worker 
Characteristics  Ratings  booklet,  and  the  trait  factors  are  presented  In 
Section  III.  As  with  the  demographic  questionnaire,  some  of  the  Items 
included  were  intended  to  aid  analysis  of  obtained  data  (such  as  Section  X 
which  obtained  Information  on  the  raters). 
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The  general  performance  ratings  (Section  I,  Job  Performance  Data)  provided 
for  the  rater's  evaluation  of  the  worker's  current  job  performance  (Item  1), 
potential  for  performance  at  the  next  grade  level  (Item  3),  supervisory 
capability  (Item  4),  and  managerial  capability  (Item  5).  Ratings  were  made  on 
a  9-polnt,  Likert-type  scale,  with  anchors  given  at  each  point.  In  addition 
to  these  items,  the  rater  was  asked  to  Indicate  whether  the  Incumbent  was  a 

supervisor  (Item  2)  and  to  estimate  the  pay  schedule  (Item  6)  and  grade  level 

(Item  7)  the  worker  would  ultimately  reach  In  Federal  service.  The  latter 
estimate,  when  combined  with  the  worker's  current  grade  level  and  pay 
schedule,  gave  the  rater's  estimate  of  the  worker's  potential  (l.e.,  how  much 
higher  that  person  would  rise  In  the  system). 

The  trait  factors  (Section  III)  were  chosen  on  the  basis  of  three 
criteria:  (a)  generallzablllty  (each  trait  should  apply  to  some  degree  to 

most  civilian  jobs  in  the  Air  Force),  (b)  comprehensiveness  (the  most 
important  traits  for  most  jobs  should  be  Included),  and  (c)  communication  (the 
trait  should  be  clearly  definable  within  two  typed  lines).  The  number  of 
traits  was  limited  to  12  since  these  traits  were  going  to  be  used  In  the 
ipsatlve  ranking  portion  of  the  project  (see  "Ipsatlve  Data*  section,  below) 

as  well  as  for  development  of  normative  ratings.  For  the  Ipsatlve  rating 

task,  12  traits  were  considered  the  maximum  number  most  raters  could 
rank-order.  As  In  Section  I,  the  9-point,  Likert-type  scale  was  selected  for 
the  rater's  response.  Sections  I  and  III  provide  normative  data. 

Rater  Accuracy  Data 


The  second  item  of  major  Interest  was  rater  accuracy  data.  Hulllns  and 
Force  (1962),  Mullins,  Seldllng,  Wllbourn,  and  Earles  (1979),  and  Cummings 
(1980)  have  demonstrated  that  rater  accuracy  can  be  determined  and  that 
accuracy  is  generalizable  across  different  rating  dimensions.  The  rating 
portion  of  the  CPAS  thus  Included  an  attempt  to  replicate  and  extend  these 
findings  and  determine  their  applicability  to  operational  settings. 

Sections  II  and  IV  were  designed  to  provide  basic  research  Information 
needed  to  calculate  the  rater  accuracy  Indices.  Section  II  consisted  of  a 
single  Item  that  required  the  rater  to  estimate  the  worker's  score  on  the 
vocabulary  (word  knowledge)  portion  of  the  CPE.  The  rater  was  told  that  for  a 
standard  administration  of  the  vocabulary  test  to  Federal  Service  employees, 
scores  generally  ranged  from  6  to  30  Items  correct,  with  an  average  score  of 
15  Items  correct.  The  rater  was  then  required  to  estimate  the  worker's 
obtained  score. 

Section  IV  consisted  of  eight  general  abilities.  Identical  In  format  to 
the  12  trait  factors  In  Section  III.  These  abilities  reflected  7  of  the  11 
CPE  subtests  (arithmetic  computation,  word  knowledge,  data  Interpretation, 
electronics  Information,  shop  practices,  mechanical  information,  and  reading 
comprehension)  and  the  Score -Checking  Test.  Four  CPE  subtests  (arithmetic 
reasoning,  hidden  figures,  pattern  comprehension,  and  verbal  analogies)  were 
not  reflected  In  the  list  of  g  .eral  abilities.  The  A  FURL  research  team 
judged  these  concepts  too  abstract  to  be  described  In  simple  terms.  It  should 
be  noted  that  the  word  knowledge  Index  was  Included  as  an  exact-score 


estimation  In  Section  II  and  as  a  rating  on  a  9-polnt  scale  In  Section  IV. 
This  technique  was  designed  to  Indicate  which  measurement  device.  If  either, 
provides  the  most  useful  Information  on  rater  accuracy. 

These  ratings  of  general  abilities,  when  combined  with  each  subject's  CPE 
scores  and  SCT  score,  provided  the  rater  accuracy  Indices.  The  Indices,  in 
turn,  were  designed  to  answer  some  Important  questions  about  rater  accuracy  In 
an  operational  setting:  (a)  Is  rater  accuracy  Indeed  a  generalizable  ability? 
(b)  Are  these  reasonably  easy  and  accurate  methods  of  determining  the  more 
accurate  and  less  accurate  raters?  (c)  Are  there  reliable  correlates  of  rater 
accuracy?  and  (d)  Can  rater  accuracy  data  be  used  In  an  operational  appraisal 
and  promotion  system?  Because  of  time  constraints  Imposed  to  operationalize 
CPAS,  the  data  were  not  analyzed  and  the  answers  to  these  issues  are  not 
contained  In  this  paper. 

Ipsatlve  Data 


The  traditional  Inflation  of  performance  ratings  in  operational  settings 
certainly  Indicates  that  novel  approaches  to  evaluation  which  might  inhibit 
Inflation  should  be  thoroughly  Investigated.  One  such  technique,  with  a 
strong  logical  appeal.  Is  the  Ipsatlve  ranking  technique.  Under  the  Ipsatlve 
technique,  the  rater  ranks  the  traits  In  a  given  list  according  to  which  are 
the  worker's  best  traits.  A  similar  ranking  Is  obtained  for  the  worker's  job 
position  (l.e.,  which  traits  are  the  most  Important  for  successful  job 
performance).  The  two  sets  of  rankings  are  then  combined  to  yield  a 
job-worker -match  coefficient.  This  coefficient  can  be  used  In  combination 
with  other  rating  and  background  data  as  a  possible  Index  of  promotion 
potential. 

A  previous  study  of  the  job-worker -trait-match  coefficient  (Mullins, 
Weeks,  &  Wllbourn,  1978)  showed  only  limited  value  of  the  Index  as  a 
predictor.  However,  this  technique  Is  new  and  deserves  further 

consideration.  This  Is  particularly  true  in  the  context  of  performance 
ratings,  where  Inflation  has  been  a  perennial  problem. 

Sections  V,  VI,  VIII,  and  IX  provided  the  scales  needed  to  compute  two 
different  job-worker -match  Indices.  Section  V  required  the  rater  to  rank  the 
12  trait  factors  (presented  as  normative  ratings  in  Section  III)  according  to 
how  well  they  applied  to  the  worker.  Section  VIII  required  a  ranking  of  these 
same  traits  according  to  how  Important  they  were  to  successful  performance  of 
the  worker's  job.  Section  VI  required  the  rater  to  rank  the  eight  general 
abilities  (presented  as  normative  ratings  in  Section  IV)  In  terms  of  the 
worker,  while  Section  IX  required  a  ranking  of  the  abilities  In  terms  of  the 
job.  The  Information  from  these  four  sections  was  to  be  combined  to  compute  a 
job-worker -trait-match  and  a  job-worker-abllity  match.  The  Ipsatlve  rankings 
could  also  be  combined  with  the  normative  ratings  to  yield  a  variety  of 
Ipsatlve-normatlve  measures.  For  example,  the  six  traits  most  Important  for 
successful  job  performance  could  be  Identified  by  the  rankings,  and  only  these 
six  normative  ratings  would  be  used  as  predictors.  Similarly,  Mullins  and 
Weeks  (1979)  provided  a  technique  for  the  "normative  use  of  Ipsatlve 
ratings."  As  will  be  discussed  later,  the  Ipsatlve  scales  were  later  removed 
from  possible  use  In  the  operational  CPAS  for  management  reasons. 
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Additional  Measures 


Section  VII  provided  two  types  of  ratings  which  were  not  so  firmly 
grounded  In  previous  research  as  the  measures  already  discussed  but  which 
could  provide  valuable  predictive  information.  Items  1  through  4  provided  for 
information  on  the  worker's  on-the-job  drive  and  productivity  level:  speed  of 
work,  productivity,  percent  time  spent  working,  and  initiative.  Each  rating 
was  made  on  a  5-point,  Likert-type  scale,  with  anchors  given  at  each  point. 
Items  5  through  21  provided  for  the  rater's  assessment  of  the  worker's  desire 
for  a  promotion.  Each  item  required  the  rater  to  indicate  (yes/no)  whether 
the  worker  would  be  willing  to  incur  a  certain  cost  in  order  to  be  promoted  to 
the  next  grade  level.  Such  costs  included:  entering  a  training/retraining 
program,  moving  to  another  organization,  accepting  a  job  outside  the  worker's 
current  job  series,  assuming  additional  duties,  and  performing  additional 
supervisory  tasks.  These  ratings  served  two  important  functions.  First,  both 
types  of  measures  were  potential  predictors,  either  as  Individual  items  or  as 
summated  measures.  Second,  the  "desire  for  promotion"  measures  (Items  5 
through  21)  provided  an  additional  index  of  rater  accuracy.  In  the 
Demographic  Questionnaire,  the  worker  indicated  his  own  desire  for  a  promotion 
on  these  same  17  items.  By  comparing  the  rater's  responses  with  the  worker's 
responses,  It  could  be  possible  to  determine  how  well  each  rater  knew  the 
worker's  desire  for  a  promotion. 

Section  X  contained  20  background  items  that  related  to  the  rater  and  to 
that  person's  familiarity  with  the  Individual  being  rated.  These  Items  were 
designed  to  serve  as  possible  control  measures  for  prediction  purposes  and  to 
be  studied  as  correlates  for  research  purposes,  particularly  for  the 
investigations  of  rater  accuracy  and  possible  adverse  Impact.  These  data  were 
collected  but,  again  because  of  subsequent  time  pressures  to  operationalize 
CPAS,  the  analyses  were  not  completed. 

IV.  PRETESTS  OF  THE  CIVILIAN  POTENTIAL  APPRAISAL  SYSTEM 
DATA  COLLECTION  INSTRUMENTS 

Initial  pretests  of  early  versions  of  the  Demographic  Questionnaire-Worker 
and  the  Job/Worker  Characteristics  Ratings  booklet  were  conducted  at  Lackland 
AFB  (N  *  15)  and  Randolph  AFB  (N  ■  20).  The  pretests  provided  information  on 
the  readability  of  the  materials,  ease  of  following  instructions,  and 
difficulties  in  using  the  scales.  Initial  feedback  on  the  booklets  led  to 
numerous  wording  changes,  changes  in  some  rating  scales,  addition/deletion  of 
some  items,  and  changing  the  order  of  some  sections.  However,  the  content  of 
the  sections  remained  substantially  the  same,  and  no  sections  were  added  or 
deleted. 

Full-scale  pretests  of  the  CPAS  data  collection  materials  were  conducted 
at  McClellan  AFB  (February  and  March  1979)  and  Patrick  AFB  (May  1979).  All 
three  parts  of  the  full  system  (testing,  demographics,  and  ratings)  were  used 
In  both  pretests.  Workers  completed  the  CPE,  SCT,  and  Demographic 
Questionnaire-Worker  in  a  single,  half-day  testing  session.  Approximately  1 
month  after  the  testing  session,  the  rating  booklets  were  mailed  out  to  the 
prospective  raters.  In  these  pretests,  three  raters  were  used  for  each 


employee:  the  supervisor,  a  co-worker,  and  an  associate  of  the  supervisor. 
(The  supervisor's  associate  was  an  employee  of  approximately  the  same  grade 
level  and  in  the  same  organization  as  the  supervisor,  who  was  familiar  with 
the  employee's  job  performance.)  Completion  of  each  rating  booklet  required 
approximately  1  hour. 

Complete  testing  /demographic  data  and  a  complete  set  of  peer,  supervisor, 
and  alternate  supervisor  ratings  were  collected  for  266  workers  at  McClellan 
AFB  and  102  workers  at  Patrick  AFB.  Means  and  standard  deviations  calculated 
on  all  measures  Indicated  that  there  were  no  serious  Inflation  or  restriction 
In  range  problems,  although  the  means  of  some  Items  were  fairly  high. 
Correlation  matrices  Indicated  that  practically  all  of  the  key  rating 
variables  (general  performance  measures,  trait  measures,  and  ability  measures) 
were  significantly  (p<,05)  Intercorrelated,  and  In  some  Instances  the 
correlations  were  quite  high.  The  drive/productivity  measures  correlated 
moderately  with  the  performance,  trait,  and  ability  ratings,  whereas  the 
promotion  desire  measures  showed  considerably  less  relation  with  the  other 
rating  Items. 

Follow-up  Interviews  at  both  bases  resulted  in  relatively  few  comments 
about  the  workability  of  the  materials;  most  comments  concerned  the  CPAS  In 
general.  A  few  people  found  the  rank-ordering  procedure  confusing;  others 
found  It  difficult  to  rank-order  the  12  traits  (Sections  V  and  VIII) 
accurately.  A  few  raters  complained  that  the  booklet  took  too  long  to 
complete,  though  most  raters  agreed  that  It  took  approximately  1  hour.  In 
general,  most  raters  found  the  rating  booklet  straightforward  and  easy  to  work 
with. 


One  problem  encountered  In  the  rating  procedure  was  obtaining  ratings  from 
the  supervisor's  associates.  Many  supervisors  did  not  have  co-workers  of 
approximately  the  same  level  In  their  organization.  Furthermore,  many  of  the 
associates  that  were  available  were  relatively  unfamiliar  with  the  employee  to 
be  rated.  Therefore,  the  supervisor's  alternate  was  substituted  for  the 
associate  as  an  additional  rating  source.  The  supervisor's  alternate  Is  a 
subordinate  of  the  supervisor  who  would  normally  assume  supervisory  duties  In 
the  supervisor's  absence.  The  Interviews  at  McClellan  AFB  and  Patrick  AFB 
Indicated  that  the  alternate  was  already  designated  or  could  easily  be 
Identified  In  most  cases.  Furthermore,  the  alternate  would  normally  be 
familiar  with  the  employee's  work;  In  some  Instances,  the  alternate  may  be 
even  more  familiar  with  the  employee  than  Is  the  supervisor.  Therefore,  the 
supervisor's  alternate  was  proposed  for  replacing  the  supervisor's  associate 
In  the  CONUS-wlde  data  collection  effort. 

The  McClellan  AFB  and  Patrick  AFB  follow-up  Interviews  also  Indicated  that 
supervisors  and  employees  tended  to  feel  the  evolving  CPAS  system  was 
preferred  over  the  current  system.  Some  56  percent  of  the  McClellan  AFB 
participants  felt  an  appraisal  system  of  objective  tests,  comprehensive 
ratings,  and  multiple  raters  provided  a  better  system,  whereas  18  percent  felt 
It  would  be  worse.  At  Patrick  AFB,  61  percent  perceived  It  as  a  better 
system,  and  30  percent  said  It  was  no  better.  Several  participants 
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particularly  favored  having  multiple  raters  where  someone  besides  their 
supervisor  could  give  ratings. 

Minor  modifications  were  made  in  all  three  components  of  the  CPAS,  In 
preparation  for  the  CONUS-wlde  test.  The  reading  comprehension  and  verbal 
analogies  subtests,  which  had  not  been  in  the  original  CPE,  were  added,  and  a 
single  mechanical  Information  subtest  was  created  by  combining  the  existing 
general  mechanics  and  mechanical  principles  subtests.  Minor  modifications 
were  made  in  the  ratings  booklet,  although  the  sections  remained  substantially 
the  same.  The  list  of  formal  training  sources  (Items  68  through  102)  was 
added  to  the  Demographic  Questionnaire-Worker,  and  the  order  of  most  of  the 
items  in  the  questionnaire  was  changed  to  present  a  more  logical  sequence  to 
the  examinee.  With  these  modifications,  the  CPAS  materials  were  prepared  for 
review  and  approval  by  the  Air  Force  Directorate  of  Civilian  Personnel 
(USAF/MPK) . 


V.  MANAGEMENT  REVIEW 

In  October  1979,  the  entire  proposed  Civilian  Potential  Appraisal  System 
was  reviewed  by  the  Air  Force  Director  of  Civilian  Personnel  and  his  staff 
(USAF/MPK).  The  proposed  system  consisted  of  the  CPE  (Including  all  11 
subtests),  SCT,  Demographic  Questionnaire-Worker,  and  the  Job/Worker 
Characteristics  Ratings  booklet  to  be  completed  by  three  raters  (supervisor, 
co-worker,  and  supervisor's  alternate)  on  each  employee.  The  ratings 
(particularly  the  supervisor's  ratings)  and  the  Demographic 
Questionnaire-Worker  were  designed  to  be  the  primary  data  sources  In  this 
system.  The  CPE  and  SCT  were  designed  to  provide  additional  Information  not 
readily  available  to  the  raters,  limit  the  Inflatablllty  of  the  system,  and 
allow  computation  of  the  rater  accuracy  measures.  The  co-worker  ratings  and 
supervisor's  alternate  ratings  were  designed  to  provide  additional  Information 
and  help  limit  Inflation  of  the  ratings. 

The  primary  concerns  expressed  by  USAF/MPK  related  to  the  legal 
acceptability  of  the  various  components  of  the  proposed  system  and  the  number 
of  man-hours  Involved  In  collecting  the  data.  As  a  result,  the  CPE,  SCT,  and 
ratings  by  the  co-worker  and  the  supervisor's  alternate  were  all  eliminated 
from  consideration  In  the  operational  system.  The  tests  were  eliminated 
because  of  the  current  controversy  surrounding  the  use  of  paper-and-pencll 
tests  for  selection/promotion  purposes  (see  United  States  Office  of  Personnel 
Management,  1979).  Despite  a  wealth  of  validity  Information,  the  Carter 
administration  had  just  signed  a  consent  decree  to  eliminate  the  PACE 
examination  for  Civil  Service  entrants  because  of  differences  In  performance 
between  minority  and  majority  applicants.  (The  PACE  Is  an  aptitude  test 
similar  to  the  CPE.)  With  the  Government's  decision  not  to  defend  even  a 
demonstratively  valid  test,  It  was  felt  the  Air  Force  should  not  undertake  the 
use  of  a  similar  test.  This  was  despite  the  fact  that  some  of  the  most  useful 
variance  In  ranking  promotion  ellgibles  In  the  trades,  administrative  job 
series,  and  some  technical  job  series  might  come  from  such  a  test.  Although 
the  tests  may  prove  to  show  little  adverse  Impact  and  have  high  validity  for 
certain  job  families,  their  use  was  a  risk  that  management  was  unwilling  to 
take.  A  second  factor  In  the  decision  to  eliminate  the  tests  was  the  number 
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of  man-hours  Involved  In  testing.  Full  testing  would  require  approximately  4 
hours  (or  more)  per  employee.  This  amount  was  prohibitive  for  even  full-scale 
testing  In  the  CONUS  data  collection  effort.  The  co-worker  ratings  and 
supervisor's  alternate  ratings  were  eliminated  from  operational  CPAS 
consideration  because  of  the  uncertainties  associated  with  use  of  these 
Individuals  as  official  raters  for  the  employee.  Operational  use  of  ratings 
by  Individuals  who  are  not  official  raters  could  create  serious  legal, 
administrative,  and  morale  problems  for  the  promotion  system.  Therefore, 
these  rating  sources  were  eliminated  from  further  consideration. 

Although  these  elements  were  deleted  from  a  possible  operational  system, 
they  were  retained  to  some  extent  for  research  purposes.  It  was  determined 
that  up  to  10  percent  of  the  CONUS  sample  could  receive  the  CPE  and  SCT,  In 
addition  to  the  other  materials.  This  would  permit  an  examination  of  rater 
accuracy,  test  validity,  and  evaluation  of  possible  adverse  Impact.  In 
addition,  the  co-worker  ratings  were  retained  for  research  purposes  only. 
This  arrangement  would  provide  a  comparison  of  co-worker  versus  supervisor 
rater  accuracy  and  provide  an  additional  rating  source  for  possible  validity 
studies.  The  supervisor's  alternate  ratings,  however,  were  eliminated 
entirely  from  the  research  design.  Seven  data  collection  alternatives  were 
presented  to  the  user  (USAF/MPK)  along  with  assessments  of  costs  (In  terms  of 
manhours)  and  research  Impact.  The  options  are  summarized  in  Table  B-3  and 
are  arranged  from  most  to  least  preferred  by  the  AFHRL  developers.  The 
options  ranged  from  the  original  research  design  that  Included  CPE  and  SCT 
tests  and  demographic  questionnaires  administered  to  20,000  target  employees 
and  60,000  raters  (one  supervisor,  one  peer,  and  one  supervisor's  alternate 
rater  for  each  target  employee)  to  no  field  test  data  collection.  The  most 
extensive  option  would  provide  an  extensive  data  base  for  future  research  and 
development  (R&D),  highly  flexible  options  for  the  operational  CPAS,  and  a 
legally  defensible  CPAS  development  process.  The  least  preferred  option  —  no 
field  test  data  collection  —  would  result  In  a  totally  subjective  rather  than 
research-based  development  process  and  would  probably  not  have  been  legally 
defensible.  USAF/MPK  selected  a  combination  of  20,000  target  workers  with 
20,000  supervisor  and  20,000  co-worker  ratings,  with  a  subset  of  approximately 
2,000  of  the  workers  given  the  CPE  and  SCT  (option  3).  The  decision 
eliminated  the  use  of  tests  as  a  CPAS  component,  eliminated  the  possible  use 
of  the  Inflation-resistant  Ipsatlve  rating  scales,  and  limited  the  scope  of 
the  research  data  base. 
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VI.  CONUS-WIDE  DATA  COLLECTION  EFFORT 

Research  Design 

The  objective  of  the  CONUS-wlde  CPAS  data  collection  effort  was  to  obtain 
measures  on  20,000  civilian  employees  representing  all  23  Air  Force  job 
families,  with  complete  supervisory  ratings,  co-worker  ratings,  and 
demographic  materials.  Between  600  and  1,100  employees  were  sampled  from  each 
job  family,  with  the  exact  number  depending  on  the  size  of  the  family.  Some 
2,000  (lot)  of  these  employees  were  administered  the  CPE  and  SCT  In  formal 
testing  sessions.  Only  the  Job/Worker  Characteristics  Ratings  (supervisor) 
and  Demographic  Questionnaire-Worker  were  to  be  used  as  Inputs  for  the 
Identification  of  candidate  operational  promotion  factors,  a  system  reduction 
exercise,  and  submission  to  the  Promotion  Policy  Data  Panels  (see  Appendix  C) 
or  for  Inclusion  In  the  final  CPAS.  The  CPE,  SCT,  and  co-worker  rating  data 
would  be  analyzed  separately  to  address  RSD  rather  than  operational  issues. 

Sample  Selection 

The  sampling  plan  for  the  CONUS-wlde  test  was  a  stratified  random  sampling 
plan,  with  specified  grade  level,  sex  group,  and  ethnic  group  restrictions. 
The  Civilian  Master  Personnel  File,  APDS-C,  Including  all  Air  Force  civilian 

employees  In  the  Continental  United  States,  was  obtained  from  the  Office  of 

Civilian  Personnel  Operations  (OCPO)  In  August  1979.  The  Initial  display 
drawn  from  this  file  for  sample  selection  purposes  was  a  three-way 

distribution  for  each  job  family,  giving  the  frequency  of  each  ethnic 
group-sex  group  combination  within  each  grade  level.  In  order  to  facilitate 
sample  selection,  extremely  small  cells  were  eliminated  by  first  collapsing 
all  of  the  •smaller1'  ethnic  group  categories  (American  Indian,  Oriental, 

Aleutian,  Eskimo,  Puerto  Rican,  and  Guamanian/Hawaiian)  Into  the  Nonminority 
(■Other*)  category.  This  procedure  resulted  In  three  ethnic  group 
categories:  Black,  Hispanic  Surnamed,  and  Other.  Next,  the  grade  levels  with 
smaller  frequencies  were  collapsed  so  that  each  grade-level  category  would 
have  a  target  frequency  of  at  least  100.  This  procedure  was  generally 
successful,  although  some  grade  categories  remained  with  fewer  than  100 
cases.  It  was  believed  that  further  collapsing  would  make  the  grade-level 
span  of  these  categories  too  large,  so  they  were  left  with  the  smaller 

frequencies.  This  problem  occurred  most  often  at  the  upper  grade  levels.  The 
result  of  this  process  was  frequency  distributions  for  three  ethnic 

categories,  two  sex  categories,  and  from  4  to  12  grade  categories  per  job 

family. 

The  number  of  cases  to  be  sampled  from  each  cell  was  based  on  these 
distributions,  according  to  the  following  rules:  (a)  the  number  of  cases 
drawn  from  each  job  family  would  vary  from  600  to  1,100,  depending  on  the 
population  of  the  family,  (b)  the  number  of  cases  from  each  grade  category 
would  be  proportional  to  that  category,  (c)  the  two  minority  ethnic  group 
categories  (Black  and  Hispanic  Surnamed)  would  be  oversampled  In  almost  every 
Instance,  and  (d)  females  would  be  oversampled  In  most  cases,  although  males 
were  also  oversampled  In  some  Instances  where  job  families  were  traditionally 
staffed  by  females.  The  ethnic  categories  and  sex  categories  were  oversampled 

In  order  to  ensure  adequate  representation  for  those  groups  that  ware 
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relatively  small  so  that  sufficient  cases  would  be  available  for  the  adverse 
Impact  analyses.  The  oversampling  procedures  were  complex  and  (necessarily) 
varied  from  job  family  to  job  family.  In  some  job  families  (e.g.,  Job 
Families  2  and  3;  Appendix  A  lists  the  job  families  and  their  respective 
specialties) ,  In  which  there  were  very  few  minority  members,  all  the  minority 
members  were  sampled.  In  many  Instances  (e.g..  Job  Families  1  and  5),  the 
minority  members  were  either  over  sampled  by  a  factor  of  two  or  sampled  until 
100  members  were  obtained,  whichever  value  was  higher  (e.g..  Job  Families  4 
and  8).  In  some  families,  this  arbitrary  value  of  100  was  raised  to  150  or 
200,  depending  on  the  size  of  the  total  sample  and  the  proportion  of  minority 
members  In  the  population.  Females  were  often  oversampled  to  ensure  adequate 
representation;  however,  in  some  cases  (e.g..  Job  Families  16  and  17),  It  was 
necessary  to  oversample  males  to  ensure  their  representation.  Thus,  the 
oversampling  procedures  were  designed  to  ensure  an  adequate  representation  of 
the  smaller  categories  and  were  based  on  a  mixture  of  objective  rules  and 
careful  case-by-case  judgments. 


67 


CO  *—  P-* 
r-  O  N 

r-  o  ro 


M  r-  O' 
tO  O 
CM  CO  ^0 


O  CM  00 
CM  O  vO 

ro  o  *<■ 

CM 


M3  O  CO 
•  •  • 
r—  ro  CM 


CM 

•  CO  to 
CM  O' 


o 

•  to  »— 
cm  10  to 


'•— *  h*-  CO 

O'  •  • 

•  Nf  U) 

ro  »—  •— 


'O  CO  N 
M>  »—  CO 


— -  O  to 
co  •  • 

•  r~  O 

ro 

CO  M3  CO 

co  so  MP 


O  to  r— 
•  •  • 
to  LO  CM 


r—  O  M3 
•  •  • 
r—  stf-  to 


o  r^ 

•  *  • 

CM  tf>  *4- 


CM 

CM 

r- • 

to 

ro 

p- 

CM 

CM 

CO 

to 

CM 

o 

vO 

ro 

ro 

*4* 

ro 

CM 

*4" 

CM 

co 

O 

CO 

ro 

CO 

s# 

to 

to 

vO 

»— 

O 

O 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

CM 

CO 

CO 

r— 

ro 

CM 

CM 

O' 

to 

ro 

O' 

CO 

to 

■— ’ 

*— 1 ’ 

w 

w 

— 

w 

■*— 1 ’ 

<w 

O 

CO 

O0 

O 

o 

O' 

rv. 

Pv 

•o 

r— 

ro 

-4- 

M3 

NT 

CO 

to 

CM 

CM 

sO 

M3 

ro 

to 

*4- 

r— 

CM 

r— 

N  N  « 
•  •  • 
to  O'  co 


CO  CO  oo 
to  IO  CM 


CO  CM  CO 
•  •  ♦ 
r—  r^.  to 


r—  r-  O' 

to  to  CM 


O'  N  O 
•  ♦  • 
ro  O'  co 


to  flO  *♦ 
M3  |v.  •# 


r-  •  sO 

•  O  • 
to  r-  O' 


co  »“*  to 
'-'O'# 


ro  »-  cm 
a  N  N 


o  00  to 
to  f—  CM 
#  N  U) 

o 


tv.  to  O' 
•  •  • 
r-  »  CO 


*4-  to  «# 

00  00  'O 


01  CL 
r-  E 
CL  <0 
C  E  CO 

•2  $  "O 
4-»  0> 

10  4J  C 
0)  f- 
3  D»  <0 
a  l  l> 

O  40  -a 
G.  h  O 


m>  ro  m> 
CM  CM  CM 


ro  O  to 
O'  s#  co 

CM  r— 


CM  tO  N 
sO  to  to 


>C  CM  O' 

o*  ro  co 
so  ro 


0)  CL 
r-  K 
Q.  10 
c  g  oo 

•2  a  x, 
+->  0) 
IO  4J  C 
f—  (i)  *4" 

3  C7)  <0 

a  l  4^ 

O  «  J3 

a.  r-  o 


s  ro  o> 

P*.  IO  to 


4-  O'  # 

'#  rv.  r- 

»—  ro  ro 

CM 


*o  o  o 

»—  CM  CM 


SO  CM  rv. 

Nt-  *#  o 


a.  40 
c  E  to 

•2  $  *o 

■M  Q> 

40  +>  C 

♦—  a>  •»- 

3  O  *0 

a  l  4J 

O  40  XI 

a  h  o 


in  vo  co 

>o  «#-  -#■ 


r-  to  CO 
co  r>.  o 
n  ro  CM 


f  to  U) 

CM 


to  to  SO 
CM  CM  co 

O'  r— 


<U  Q. 
r—  E 
CL  40 

c  e  </) 

•2  <5?  *o 

+J  4U 

40  K 

•—  0)  **“ 
3  cn 
a  l  4J 

0*0X1 

a.  H  o 


r~  O'  r— 
to  ro  M- 


co  to  to 
00  to  O' 
*—  CM  *— 


O' 

ro 

to 

• 

• 

CM 

vO 

ro 

CM 

CM 

w 

*4- 

vO 

*4* 

0) 

M3 

M- 

CM 

c 

Pv 

t- 

c 

o 

v> 

at 

L. 

0> 

QJ 

a. 

CL 

r— 

E 

Cl 

40 

c 

E 

to 

>>  40  4J  c 

r-  r-  0)  •*“ 

f-  3  cr» 

E  ex  J-  +j 

(Q  O  «J  X) 

u_  a  h  o 


r 


*o 

LO 

CO 

O 

o 

NO 

CO 

O' 

O 

ro 

CM 

CM 

o 

p— 

CO 

<o 

CM 

ro 

o 

ro 

CO 

O 

in 

CM 

CO 

CO 

M> 

r*-. 

O 

u> 

'4- 

o 

t"- 

to 

tf> 

CO 

r— 

CM 

•4 

p— » 

.—•* 

O 

to 

— «. 

o 

ro 

r— 

o 

r** 

• 

• 

O 

• 

o> 

CM 

• 

NO 

• 

• 

• 

• 

ro 

ro 

• 

ro 

• 

• 

o 

• 

CM 

N 

fs. 

u> 

*— 

r— 

o 

CM 

»— 

O' 

5 

CO 

P» 

o 

NO 

CO 

CM 

CO 

ro 

CM 

u> 

ro 

CO 

NT 

o 

*4 

O' 

CO 

r-. 

•4- 

O' 

CO 

u> 

• 

O' 

-* 

o 

ro 

,—s 

«— » 

f 

• 

• 

o 

CO 

*4- 

r— 

P'S. 

*4 

*— * 

*•— * 

• 

ro 

"4* 

• 

• 

• 

• 

• 

• 

*4 

CO 

NO 

p 

*—• 

CM 

*4* 

NO 

»— 

ro 

ro 

• 

• 

• 

o' 

"W» 

N»^ 

*x^ 

»•»«* 

N^ 

•*-+ 

O 

r* 

P™ 

N— ’ 

"n-p 

> 

NO 

NO 

vl* 

NO 

CM 

CM 

CO 

4 

O 

CO 

P*“» 

CO 

*4 

CM 

CM 

r— * 

CO 

*4 

O' 

»  N  O 
•  •  • 
r—  K> 


O'  CO  *4 

•  i  « 

N  CO  CO 


r»  n*.  CM 

•  •  • 
N  N  O 


nO 

00 

ro 

*4 

O' 

o 

00 

CM 

p— 

NO 

r— 

4 

O' 

ro 

CM 

to 

4 

O' 

NO 

CO 

p-. 

4 

CO 

NO 

ro 

_ 

CM 

NO 

.»«•* 

«— » 

r— 

• 

• 

o 

r— 

O' 

NO 

to 

O 

© 

O' 

fN. 

• 

'4- 

CM 

• 

• 

• 

• 

• 

• 

• 

• 

• 

NO 

p— 

ro 

O' 

CO 

4 

ro 

ro 

CM 

NO 

NO 

4 

ro 

CM 

*4* 

ro 

fs. 

ro 

CM 

r— 

to 

r— 

P— 

P'S. 

o 

nO 

ro 

CM 

p— 

n4 

ro 

ro 

•»*» 

4 

#—» 

p— 

NO 

CO 

ro 

to 

o 

CM 

NO 

NO 

fNn 

O' 

"4* 

• 

• 

• 

• 

• 

■ 

• 

• 

• 

• 

• 

• 

O' 

4 

4 

C0 

to 

^4 

CO 

P— 

NO 

ro 

ro 

M3 

p— 

p— 

u> 

4 

>4 

NO 

*4 

*4 

C 

fs. 

•4 

*4 

N»-» 

o 

«w » 

to 

•p- 

o 

O' 

rs. 

fN. 

O 

O 

to 

r— 

O' 

ro 

44 

CO 

nO 

*F“ 

r— 

p— 

CO 

ro 

p~ 

O' 

*4 

o 

<0 

r~ 

*4 

CO 

c 

O 

p" 

r-* 

to 

ro 

10 

ro 

CM 

CM 

L 

O 

ro 

CM 

jc 

O' 

o 

p— 

44 

ro 

u 

•p- 

«/» 

01 

c 

•r* 

h- 

•5 

C 

*r* 

0) 

E 

o 

c 

h 

3 

4 

O' 

NO 

*4* 

o 

PN* 

o 

c 

r^. 

CM 

CO 

N 

to 

O' 

CO 

c 

• 

• 

• 

r* 

• 

• 

• 

o 

• 

• 

• 

44 

• 

• 

• 

*r- 

to 

NO 

P". 

CO 

CO 

O' 

•r- 

o 

to 

r*-. 

C 

o 

ro 

U. 

NO 

4 

"4 

o 

CM 

p^ 

p— 

44 

CM 

CM 

CM 

4) 

CM 

ro 

ro 

V»* 

+—* 

*T» 

N>-P 

N»«* 

«0 

E 

*o 

c 

€) 

tt> 

c 

O' 

O 

O' 

■5 

ro 

o* 

NO 

&- 

CM 

p* 

o> 

Pn. 

ro 

to 

CM 

CO 

PN« 

O' 

o 

ro 

U 

CM 

to 

CM 

to 

ro 

to 

co 

•4* 

ro 

CM 

4) 

ro 

CM 

p— • 

0> 

»4 

c 

co 

CM 

p" 

g 

ro 

h- 

4 

oc 

£ 

•r- 

v» 

*o 

4-» 

w 

tz 

C 

4) 

to 

<0 

3 

C 

t- 

O 

tfi 

4* 

O 

0) 

</> 

4) 

t 

4) 

C 

4> 

o 

a 

3 

p— • 

< 

4> 

CL 

41 

CL 

< 

4) 

CL 

0) 

CL 

p— • 

E 

E 

E 

f— 

E 

o. 

<0 

Ol 

to 

O. 

ro 

CL 

to 

•  • 

c 

E 

to 

•  • 

C 

E 

CO 

•  • 

C 

E 

to 

•  • 

c 

E 

to 

o 

o 

<2 

p"* 

O 

<7! 

CM 

O 

a 

ro 

o 

to 

r— 

T* 

-o 

r- 

*p“ 

■o 

•p- 

T3 

t* 

to 

■o 

44 

4) 

44 

4> 

44 

4) 

44 

4) 

>» 

« 

44 

c 

>» 

to 

44 

c 

>» 

« 

44 

C 

>» 

to 

44 

c 

p“^ 

P“ 

0) 

*p» 

r“" 

4) 

•f* 

p— 

P-* 

4) 

*T" 

p— 

p* 

4> 

*p“ 

•r- 

3 

Ol 

<d 

*r* 

3 

o» 

« 

•p- 

3 

O) 

<0 

•p* 

3 

o» 

<0 

■ 

CL 

u 

44 

E 

CL 

J- 

44 

B 

CL 

L. 

44 

E 

o. 

L. 

44 

tO 

O 

to 

X3 

« 

O 

to 

JQ 

<0 

o 

O 

^3 

<0 

O 

tO 

.O 

u_ 

CL. 

r- 

O 

U. 

O. 

H 

O 

u. 

CL 

r- 

O 

u. 

CL 

r- 

o 

Selection  of  the  target  worker  sample  was  as  follows: 

1.  The  number  of  cases  to  be  drawn  from  each  grade-level,  sex-category, 
and  ethnic-category  cell  was  specified.  Job  series,  major  command,  location 
and  time-in-service  characteristics  of  the  sample  were  not  specified  In  the 
sample  selection  process  and  thus,  were  random.  The  total  number  of  cases  In 
the  population  and  samples  by  subgroup  category  are  given  In  Table  B-4. 

2.  A  total  of  74  CONUS  bases  were  sampled.  All  Air  Force  bases  with  less 
than  500  civilian  employees  were  eliminated  from  consideration  In  sampling.  In 
order  to  reduce  the  amount  of  low-yield  administrative  work.  McClellan  AFB, 
Patrick  AFB,  and  Norton  AFB  were  also  eliminated  from  consideration,  due  to 
the  extensive  pretesting  of  either  CPAS,  the  General  Manager  Appraisal  System 
(GMAS),  or  the  Job  Performance  Appraisal  System  (JPAS)  at  these  bases. 

3.  In  order  to  ensure  availability  of  subjects  In  target  categories, 

should  target  workers  become  unavailable,  the  target  employees  were  paired  or 
matched  with  an  alternate  employee  at  the  same  Air  Force  base  according  to 
their  job  family,  grade  level,  ethnic  group,  and  sex  category.  Target 

employees  that  could  not  be  perfectly  paired  with  an  alternate  were  still 
matched  on  grade  level  and  ethnic  group  If  possible,  or  as  a  minimum,  on  grade 

level  and  sex.  Employees  that  could  not  be  paired  on  at  least  two 

characteristics  were  retained  In  the  sample  as  "odd  lots"  without  a  match 
available. 

4.  The  required  numbers  of  pairs  for  each  job  family  and  each  category 
were  randomly  drawn  by  computer.  One  member  of  each  pair  was  randomly 
designated  as  the  "target"  worker,  and  the  other  member  became  the  "alternate" 
worker.  In  those  cases  where  only  one  worker  was  available  for  a  given  job 
family  at  a  base,  an  unmatched  alternate  was  selected.  Some  98.3  percent  of 
the  target  employees  had  a  matching  alternate. 

5.  Finally,  the  roster  of  selected  target  and  matching  alternate  workers 
was  printed.  This  list,  organized  by  base  and  job  family,  provided  the  name, 
SSAN,  organization,  and  office  symbol  for  each  target  and  alternate  worker  In 
the  sample. 

As  noted,  the  management  review  determined  that  only  10  percent  of  the 
CONUS  sample  would  receive  the  CPE  and  SCT.  Therefore,  2,000  workers  within 
the  already-obtained  sample  were  selected  for  the  "full  testing"  process 
(e.g.,  to  receive  the  CPE  and  SCT,  as  well  as  be  rated  by  their  supervisor  and 
one  peer).  Only  perfectly  matched  pairs  of  workers  and  alternates  were  used, 
and  equal  numbers  of  workers  were  drawn  from  each  ethnic-sex  category 
combination  (333  workers  from  each  of  the  six  ethnic-sex  cells).  The  pairs 
were  randomly  selected  by  computer  within  these  constraints,  preserving  the 
proper  proportion  of  subjects  from  each  job  family  but  without  regard  to  Air 
Force  base.  The  result  of  this  procedure  was  a  subsample  of  workers  (within 
the  full  sample)  that  was  selected  for  the  full  testing  process  by  stratified 
random  sampling,  on  the  basis  of  ethnic-sex  category  combination  and  job 
family. 


Administrative  Procedures 


Administration  of  the  CPAS  data  collection  tests,  questionnaires,  and 
rating  forms  at  each  base  was  handled  by  an  employee  of  the  Central  Civilian 
Personnel  Office  (CCPO)  who  had  been  designated  as  the  contact  point.  At  the 
beginning  of  the  CONUS  administration  (November  1979),  each  CCPO  contact 
received  a  full  set  of  materials:  (a)  Job/Worker  Characteristics  Ratings 
booklets,  (b)  Demographic  Questionnaire-Worker,  (c)  Civilian  Personnel 
Examinations,  (d)  Score  Checking  Tests,  (e)  personnel  Identification  forms, 
(f)  instructions  to  CCPO,  Including  general  administration  Information, 
booklets  for  administration  of  the  questionnaire  and  tests,  and  union 
notification  Instructions,  (g)  rosters  of  selected  target,  matching,  and 
second  matching  workers,  (h)  answer  sheets,  cover  letters,  envelopes,  and 
printed  return  mailing  labels,  as  needed  for  administration. 

The  data  collection  phase  was  handled  by  the  CCPOs  In  a  five-step 
process:  (a)  selection  of  workers,  (b)  Identification  and  selection  of 
raters,  (c)  administration  of  tests  and  demographic  questionnaires,  (d) 
administration  of  rating  forms,  and  (e)  collection,  checking,  and  accounting 
of  materials. 

1,  Selection  of  Workers.  The  CCPO  was  Instructed  to  select  all  target 
workers  on  the  "Roster  of  Selected  Target,  Matching,  and  Second  Matching 
Workers."  The  matching  alternate  was  selected  if  the  target  worker  could  not 
be  scheduled  for  a  testing  or  background  data  session  at  any  time  or  If  It  was 
Impossible  to  obtain  at  least  one  of  the  two  sets  of  required  ratings  on  the 
target  worker.  If  the  matching  alternate  was  also  unavailable,  the  CCPO 
randomly  selected  another  replacement  from  the  same  job  family  roster  of 
alternates.  Workers  who  were  selected  for  the  full  testing  procedure  (CPE  and 
SCT,  in  addition  to  the  other  materials)  were  Indicated  on  the  roster  and  were 
scheduled  for  an  Initial  *  l/2»hour  testing/demographics  session.  All  other 
workers  (approximately  90S  of  the  sample)  received  the  Demographic 
Questionnaire-Workers  by  mall  or  were  scheduled  for  a  half-hour  group 
administration  session  at  a  location  selected  by  the  CCPO. 

2.  Identification  and  Selection  of  Raters.  When  the  worker  had  been 
Identified,  the  CCPO  mailed  out  an  Introductory  letter,  with  the  Personnel 
Identification  Form  attached,  to  the  worker's  supervisor.  The  Introductory 
letter  briefly  described  the  nature  of  the  study  and  assured  the  supervisor 
that  all  data  collected  from  the  worker  and  the  raters  would  be  strictly 
confidential  and  used  for  research  purposes  only.  The  supervisor  was 
requested  to  release  the  employee  for  testing  or  demographics  questionnaire 
administration  at  a  specified  time  and  Informed  that  the  supervisor  and  other 
raters  would  also  be  requested  to  complete  some  survey  materials.  The 
supervisor  was  then  requested  to  Identify  his  own  alternate  and  as  many  of  the 
employee's  co-workers  as  possible,  on  the  Personnel  Identification  Form.  The 
co-workers  were  defined  as  workers  "doing  essentially  the  same  kind  of  work  as 
the  designated  employee.  These  co-workers  should  know  the  employee  well 
enough  to  rate  him/her. . .Co-workers  may  be  either  military  or  civilian." 
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The  CCPO  randomly  selected  a  co-worker  rater  from  the  list  of  co-workers 

provided  by  the  supervisor.  This  random  selection  was  designed  to  avoid 

consistently  using  the  worker's  most-,  least-  or  “typically-"  familiar 
co-worker.  Instead,  a  variety  of  different  co-worker  types  was  sampled. 

3.  Administration  of  Test  and  Demographic  Questionnaires.  Separate 

procedures  were  followed  for  the  “full -testing*  and  the 
“demographlcs/ratlngs-only"  workers.  The  demographlcs/ratlngs-only  workers 
received  their  questionnaires  either  by  mall  or  In  the  mass  survey 

administration  sessions  which  were  scheduled  depending  on  the  requirements  of 
the  local  CCPO.  The  survey  session  required  approximately  35  minutes  (15 
minutes  for  administrative  details  and  20  minutes  for  completion  of  the 
questionnaire).  Full-testing  workers  were  tested  in  a  single  A  1/2-hour 

testing  session  In  which  they  completed  the  following  Items,  In  order:  (a) 
SCT  (20  Items;  15  min),  (b)  the  Demographic  Questionnaire-Worker  (102  Items; 
approximately  20  min),  and  (c)  CPE  (223  Items;  approximately  2  1/2  hours). 
Approximately  85  minutes  were  provided  for  administrative  details,  breaks,  and 
possible  delays.  Tests  were  administered  by  the  CCPO  contact  or  local  Test 
Control  Officer  according  to  a  standardized,  administration  booklet.  The 
procedures  mirrored  those  routinely  used  for  administration  of  the  AQE-0 
(Vltola  et  al.,  1971),  from  which  the  CPE  was  adapted.  Each  subtest  was 
preceded  by  a  set  of  specifically  tailored  Instructions  and  one  or  two 
Illustrative  examples.  All  subtests  were  timed  according  to  the  limits  given 
In  Table  B-1. 

4.  Administration  of  Ratings  Forms.  The  rating  materials  were  mailed  out 
to  the  supervisor  and  co-worker  of  each  worker  within  1  month  after  the 
testing/demographic  Information  was  completed.  Each  rating  packet  Included  a 
cover  letter,  a  Job/Worker  Characteristics  Ratings  booklet,  a  corresponding 
answer  sheet,  and  a  return  envelope.  The  cover  letter  solicited  the  rater's 
participation  In  the  study,  which  was  described  as  a  study  of  “how  accurately 
people  can  rate  other  people  In  their  organization."  Raters  were  assured  that 
the  ratings  would  be  kept  strictly  confidential  and  that  they  would  have  no 
effect  on  the  rater  or  the  rated  Individual.  Raters  were  also  Instructed  not 
to  consult  with  any  other  Individuals  In  their  organization  who  may  have  been 
given  rating  materials.  Raters  then  completed  the  Identification  data  and 
proceeded  through  the  10  sections  of  the  ratings  booklet  (Appendix  B-2).  All 
ratings  were  returned  to  the  local  CCPO  In  individual,  sealed  envelopes. 

5.  Collection,  Checking,  and  Accounting  of  Materials.  CCPOs  were 
required  to  sort  and  check  materials  as  they  were  received.  Incorrect  or 
partially  completed  answer  sheets  were  returned  to  the  rater  for 
reaccomplishment.  Additionally,  each  CCPO  provided  an  "Incomplete  Data  Log," 
which  described  the  workers  with  missing  data  and  reasons  for  missing  sheets 
(e.g.,  worker  unwilling  to  participate  under  Privacy  Act  provisions,  no 
co-worker  available  for  the  rated  Individual,  supervisor  on  TOY). 

Delay  In  Data  Collection  Efforts 


Original  plans  called  for  the  data  collection  efforts  to  be  completed  by 
February  1980,  approximately  3  months  after  the  study  was  started.  However, 
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the  Air  Force  Logistics  Command  Directorate  of  Civilian  Personnel  opted  to 
delay  the  study  until  the  concerns  of  the  American  Federation  of  Government 
Employees  (AFGE)  were  resolved.  On  two  occasions,  national  representatives  of 
AFGE  had  been  previously  Informed  of  the  CPAS  development  effort  Including  a 
detailed  briefing  at  which  the  AFGE  representative  stated  that  they  would 
oppose  any  promotion  system  that  used  any  ranking  factors  other  than 
longevity.  Although  three  bargaining  sessions  were  held,  AFGE  concerns 
remained  unresolved  by  February  1980.  Therefore,  Headquarters  USAF  directed 
that  the  data  collection  efforts  In  support  of  the  CPAS  proceed  within  AFLC  as 
with  the  other  Air  Force  bases  regardless  of  the  opposition.  As  a  result,  the 
closeout  date  for  the  data  collection  effort  had  to  be  reset  to  June  1980. 
AFGE  subsequently  filed  an  Unfair  Labor  Practices  (ULP)  complaint. 

Database  Generation 

All  demographic  and  rating  answer  sheets  received  at  AFHRL  by  the  revised 
closeout  were  checked  and  entered  Into  the  CONUS-CPAS  master  file.  Numerous 
answer  sheets  were  received  after  the  closeout  date,  particularly  from  the 
AFLC  bases.  However,  the  need  to  process  the  records  for  Input  to  the 
Promotion  Policy  Data  Panel  analyses  required  that  data  received  after  June 
1980  be  left  out  of  the  database.  CPE  and  SCT  answer  sheets  were  not  Included 
In  the  file  since  they  were  not  to  be  used  In  the  CPAS  development.  These 
answer  sheets  were  retained  and  placed  with  the  co-workers'  ratings  In  a 
separate,  "research-only*  database  file.  Each  subject's  record  consisted  of 
five  parts:  (a)  master  record  Identification,  (b)  worker's  Demographic 
Questionnaire-Worker,  (c)  supervisor's  Job/Worker  Characteristics  Ratings,  and 
(e)  pre-survey  source  data. 

Field  Test  Participation  Rate 

This  section  presents  Information  on  the  data  return  rate  from  the 
CONUS-wlde  field  test.  Analyses  of  the  CONUS  data  are  presented  In  Appendix  C. 

Table  B-2  presents  the  numbers  and  percentages  of  employees  that  were 
available  for  sampling  In  each  job  family  population,  the  target  sample,  and 
the  obtained  sample.  The  figures  for  each  family  are  broken  down  by 
ethnic-sex  category.  Values  for  the  obtained  sample  reflect  only  those  cases 
for  which  complete  data  sets  were  received.  A  complete  data  set  consisted  of 
both  the  Demographic  Questionnaire-Worker  and  the  supervisor's  Job/Worker 
Characteristic  Ratings  (or  co-worker’s  ratings  that  could  be  substituted  for 
the  supervisor's  ratings). 

Complete  data  sets  were  received  for  12,865  workers,  for  a  total  return 
rate  of  64.32  percent.  The  return  rate  per  job  family  varied  from  a  low  of 
45.7  percent  (Job  Family  20)  to  a  high  of  77.7  percent  (Job  Family  5),  with  a 
median  rate  of  67.8  percent.  The  smallest  number  of  usable  cases  obtained  was 
327  In  Job  Family  2,  while  the  largest  number  was  738  In  Job  Family  17.  In 
general,  the  return  rate  was  higher  for  the  General  Schedule  families,  with  a 
range  of  54.4  percent  to  77.7  percent,  than  for  the  Wage  Grade  families,  with 
a  range  of  45.7  percent  to  68.5  percent.  Demographic  Questlonnalres-Worker 
were  received  from  15,406  workers  (77.03  percent),  while  12,978  supervisor's 
ratings  were  received  (64.89  percent),  along  with  11,597  (58.00  percent) 
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co-worker  ratings.  This  Is  a  solid  return  rate,  and  It  provided  amols  data 
for  all  fannies  for  the  variable  reduction  phase  and  the  Promotion  Policy 
Data  Panels  exercise,  fro*  which  the  final  set  of  rating  factors  and  their 
relative  weights  were  developed. 

Data  Analysis 

After  completion  of  the  database,  two  major  tasks  remained  In  the  final 
development  of  the  CPAS:  reduction  of  the  numerous  variables  to  a  manageable 
subset  and  the  Identification  and  weighting  of  the  final  operational  set  of 
variables.  Reduction  of  the  numerous  variables  to  a  manageable  subset  is 
discussed  In  the  body  of  this  technical  paper,  while  the  weighting  of  the 
operational  variables  Is  presented  In  considerable  detail  in  Appendix  C. 
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PRIVACY  ACT  STATEMENT 


AUTHORITY:  Statute  10  U.S.C.  8012;  Secretary  of  Air  Force,  Powers, 
Duties,  Delegation  by  Compensation,  E09397,  22  Nov  43,  Numbering 
System  for  Federal  Accounts  Relating  to  Individual  Persons. 

PRINCIPAL  PURPOSE:  This  information  will  be  used  solely  for  Air 
Force  Research  and  Development  purposes.  Use  of  the  Social  Security 
Account  Number  is  necessary  to  make  positive  identification  of  the 
individual's  records. 

ROUTINE  USES:  Information  provided  by  respondents  will  be  treated 
as  CONFIDENTIAL  and  will  be  used  for  official  research  purposes 
only.  Responses  to  this  survey  will  be  known  only  to  the  researchers  in 
the  form  of  group  statistics.  Your  responses  will  not  be  revealed  to 
anyone  for  any  reason.  This  includes  your  supervisor,  your  co-workers, 
any  subordinates,  your  CPO  personnel  specialists,  and  any  union 
officials.  None  of  these  individuals  will  know  how  you  answered  this 
survey.  Although  respondents  are  identified  by  name  and  SSAN,  the 
research  information  obtained  will  be  used  only  to  improve  evaluation 
and  promotion  techniques  within  the  Air  Force  Civil  Service  Personnel 
System. 

DISCLOSURE  IS  VOLUNTARY:  There  will  be  no  adverse  personnel 
actions  taken  if  you  choose  not  to  participate.  However,  failure  to 
provide  information  would  hinder  the  Air  Force's  ability  to  improve 
the  effectiveness  of  the  civilian  personnel  system.  The  personnel  system 
continues  to  improve  only  with  your  assistance  to  make  additional 
refinements  in  policies  and  procedures.  Your  cooperation  in  this  effort 
is  appreciated. 
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The  following  duestions  deal  with  your  background  and  interest  in  promotional  opportunity.  Please  answer 
them  as  truthfully  and  completely  as  you  can. 


23.  Your  si 


A.  /Male 

B.  Female 

24.  Your  age 

A.  Less  than  25  years 

B.  25  -  30 

C.  31  -  35 

0.  36  -  40 

E.  41  -45 

F  46-50 

G.  51-55 

H.  Over  55  years 

25.  Your  racial  or  ethnic  category 

A.  American  Indian  or  Alaskan  native 

B.  Asian  or  Pacific  Islander 

C.  Black,  not  of  Hispanic  origin 

D.  Hispanic 


Your  pay  schedule 

A. 

GS 

B. 

WG 

C. 

WL 

D. 

WS 

E. 

Other 

Your  grade 

A. 

1 

1. 

9 

B. 

2 

J. 

10 

C. 

3 

K. 

11 

D. 

4 

L. 

12 

E. 

5 

M. 

13 

F. 

6 

N. 

14 

G. 

7 

O. 

15 

H. 

8 

P. 

16 

30.  What  was  your  pay  schedule  when  you  first 
entered  Civil  Service? 


E.  White,  not  of  Hispanic  origin 

A. 

GS 

B. 

WG 

What  is  the  highest  education  level  you  have 

C. 

WL 

reached? 

D. 

WS 

A.  Eighth  grade  or  lower 

E. 

Other 

B.  Attended  high  school;  did  not 

graduate 

C.  Completed  high  school  with 
graduation 

D.  Attended  two  years  or  less  college 

E.  Associate  degree  or  more  than  two 

years  of  college 

F.  Bachelor's  degree 

G.  Bachelor's  degree  with  some  graduate 
work;  no  graduate  degree 

H.  Master's  degree  or  PhD 

Your  present  civil  service  status 

A.  Temporary 

B.  Career  conditional 

C.  Career 

D.  Other 


What  was  your  grade  level  when  you  first 
entered  Civil  Service? 


A. 

1 

1, 

9 

B. 

2 

J. 

10 

C. 

3 

K. 

11 

D. 

4 

L. 

12 

E. 

5 

M. 

13 

F. 

6 

N. 

14 

G. 

7 

O. 

15 

H. 

8 

P. 

16 

32.  How  long  have  you  worked  for  the  Civil 
Service? 

A.  Less  than  1  year 

B.  1-2  years 

C.  3-5  years 

D.  6  -  10  years 

E.  11-15  years 

F.  16  — 20  years 

G.  Over  20  years 

GO  ON  TO  THE  NEXT  PAGE. 
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33.  How  much  total  time  have  you  had  in  your 
present  grade  level? 

A.  Less  than  1  year 

B.  1-2  years 

C.  3 -4 years 

D.  5-6  years 

E.  7-8  years 

F.  9 -10  years 

G.  Over  10  years 

34.  How  long  have  you  worked  in  your  present 
job  series? 

A.  Less  than  1  year 

B.  1-2  years 

C.  3-4  years 

D.  5-6  years 

E.  7-8  years 

F.  9 -10  years 

G.  Over  10  years 

35.  Is  your  supervisor: 

A.  Civilian 

B.  Military 

36.  Are  you  currently  an  official  supervisor? 

A.  Yes 

B.  No 

37.  Have  you  received  any  formal 
government-administered  training  for  the 
position  you  currently  hold? 

A.  Yes 

B.  No 

38.  Have  you  received  an  Outstanding 
Performance  Rating  (OPR)  within  the  last 
year? 

A.  Yes 

B.  No 

39.  Have  you  received  a  Quality  Step  Increase 
(QSI)  within  the  last  year? 


40.  Have  you  received  a  Sustained  Superior 
Performance  award  within  the  last  year? 

A.  Yes 

B.  No 

41.  Are  you  currently  eligible  for  promotion? 

A.  Yes 

B.  No 

42.  How  many  times  have  you  interviewed  for 
promotion  within  the  last  year? 

A.  0 

B.  1 

C.  2 

D.  3 

E.  4  or  more 

43.  Have  you  been  promoted  in  the  last  year? 

A.  Yes 

B.  No 

44.  Can  you  be  promoted  to  a  higher-level 
position  in  your  current  job  series? 

A.  Yes 

B.  No 

45.  If  you  can  be  promoted  to  a  higher  position 
in  your  job  series,  is  it: 

A.  Non-supervisory 

B.  Supervisory 

C.  Could  be  either  supervisory  or 
non-supervisory 

D.  I  cannot  be  promoted  in  my  current 
job  series 

46.  Do  you  plan  to  continue  Civil  Service 
employment  until  retirement? 

A.  Yes 

B.  No 

C.  I  do  not  know 


A.  Yes 

B.  No 


GO  ON  TO  THE  NEXT  PAGE. 


'17  On  the  average,  how  many  hours  per  week  49.  On  the  average,  how  many  hours  per  week 
<Jo  you  spend  on  community  service  do  you  spend  in  extracurricular  intellectual 

activities  (little  league  coaching,  church  activities  (reading,  music,  theater,  education, 

activities,  scouts,  PTA,  Red  Cross,  etc.)?  hobbies,  etc.)? 


A. 

None 

A. 

5  or  fewer  hours  per  week 

B 

1  3  hours  per  week 

B. 

6-10  hours  per  week 

C. 

4  -  -  6  hours  per  week 

C. 

11  —  15  hours  per  week 

D. 

7-9  hours  per  week 

D. 

16  -  20  hours  per  week 

r. 

10  or  more  hours  per  week 

E. 

More  than  20  hours  per  week 

On  the  average,  how  many  hours  per  week 

50.  In  the  past  year,  how  much  effort  have  you 

do  you  spend  in  extracurricular  activities 

spent  toward  getting  a  promotion? 

that 

involve  physical  exertion  (sports. 

exercising,  yardwork,  home  maintenance. 

A. 

No  effort— 1  am  not  interested  in  a 

etc.)? 

promotion 

B. 

No  effort -but  would  like  a  promotion 

A 

5  or  fewer  hours  per  week 

C. 

A  little  effort 

B. 

6-10  hours  per  week 

D. 

A  moderate  effort 

C. 

11  -  15  hours  per  week 

E. 

A  lot  of  effort 

D. 

16-20  hours  per  week 

E. 

More  than  20  hours  per  week 

GO  ON  TO  THE  NEXT  PAGE, 
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USE  THE  FOLLOWING  FOR  QUESTIONS  51  —  67:  Most  promotions  involve  taking  on  additional  duties, 
responsibilities,  training,  and  effort  for  successful  job  performance.  Which  of  the  following  responsibilities 
or  activities  would  you  accept  for  promotion  to  the  next  grade  level? 


51.  Increase  the  amount  of  your  work? 

A.  Yes 

B.  No 

52.  Increase  the  quality  of  your  work? 

A.  Yes 

B.  No 

53.  Improve  your  working  relationships  with 
supervisor,  co-workers,  or  subordinates? 

A.  Yes 

B.  No 

54.  Assume  extra  additional  duties? 

A.  Yes 

B.  No 

55.  Take  more  initiative  on  important  projects? 

A.  Yes 

B.  No 

56.  Perform  additional  supervisory  tasks? 

A.  Yes 

B.  No 

57.  Assume  high  pressure  tasks  or  duties,  such  as 
public  briefings,  working  on  tight  deadlines, 
negotiations,  or  dealing  with  other  agencies 
on  major  projects? 

A.  Yes 

B.  No 

58.  Work  overtime? 

A.  Yes 

B.  No 

59.  Enter  a  training/retraining  program  on 
government  time  and  expense? 

A.  Yes 

B.  No 
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60.  Enter  a  training/retraining  program  on  own 
time  but  at  government  expense? 

A.  Yes 

B.  No 

61.  Enter  a  training/retraining  program  on  own 
time  and  expense? 

A.  Yes 

B.  No 

62.  Move  to  another  branch/section  within  the 
present  organization? 

A.  Yes 

B.  No 

63.  Move  to  another  organization  on  base? 

A.  Yes 

B.  No 

64.  Move  to  another  base  within  the  same  local 
area? 

A.  Yes 

B.  No 

65.  Move  to  another  base  outside  of  the  local 
area? 

A.  Yes 

B.  No 

66.  Accept  a  job  outside  of  your  current  job 
series? 

A.  Yes 

B.  No 

67.  Devote  more  time  to  administrative/ 
managerial  duties? 

A.  Yes 

B.  No 


GO  ON  TO  THE  NEXT  PAGE. 


USE  THE  FOLLOWING  FOR  QUESTIONS  68-102:  Have  you  ever  had  formal  training  in  any  of  the 
following  courses,  in  high  school,  college,  or  trade  school? 


68. 

Algebra 

77. 

Electrical  Engineering 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

69. 

Geometry 

78. 

Computer  Science 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

70. 

Trigonometry 

79. 

Physics 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

71. 

Analytical  Geometry 

80. 

Chemistry 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

72. 

Calculus 

81. 

Biology 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

73. 

Probability  or  Statistics 

82. 

Anatomy  or  Physiology 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

74. 

Business  Mathematics 

83. 

Geology 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

75. 

Mechanical  Engineering 

84. 

Geography 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

76. 

Chemical  Engineering 

85. 

Basic  Electronics 

A.  Yes 

A.  Yes 

B.  No 

B.  No 

GO  ON  TO  THE  NEXT  PAGE. 


86.  Bookkeeping/Accounting 


95.  English  Literature 


A.  Yes 

B.  No 

87.  Commercial  Graphic  Art 

A.  Yes 

B.  No 

88.  Drafting/Mechanical  Drawing 

A.  Yes 

B.  No 

89.  Radio  Repair 

A.  Yes 

B.  No 

90.  Auto  Repair 

A.  Yes 

B.  No 

91.  Mechanical/Technical  Theory 

A.  Yes 

B.  No 

92.  Speech/Drama 

A.  Yes 

B.  No 

93.  English  Grammar/Composition 

A.  Yes 

B.  No 

94.  American  Literature 

A.  Yes 

B.  No 


A.  Yes 

B.  No 

96.  World  Literature 

A.  Yes 

B.  No 

97.  Creative  Writing 

A.  Yes 

B.  No 

98.  Technical  Writing 

A.  Yes 

B.  No 

99.  Journalism 

A.  Yes 

B.  No 

100.  Introductory  Foreign  Language 

A.  Yes 

B.  No 

101.  Intermediate  Foreign  Language 

A.  Yes 

B.  No 

102.  Advanced  Foreign  Language 

A.  Yes 

B.  No 
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USAF  SCN  79-20 

(Expire*  31  Dec  79) 


AIR  FORCE  SYSTEMS  COMMAND 

BROOKS  AIR  FORCE  BASE, TEXAS  78235 


PRIVACY  ACT  STATEMENT 


AUTHORITY:  Statute  10  U.S.C.  8012;  Secretary  of  Air  Force,  Powers, 
Duties,  Delegation  by  Compensation,  E09397,  22  Nov  43,  Numbering 
System  for  Federal  Accounts  Relating  to  Individual  Persons. 

PRINCIPAL  PURPOSE:  This  information  will  be  used  solely  for  Air 
Force  Research  and  Development  purposes.  Use  of  the  Social  Security 
Account  Number  is  necessary  to  make  positive  identification  of  the 
individual's  records. 

ROUTINE  USES:  Information  provided  by  respondents  will  be  treated 
as  CONFIDENTIAL  and  will  be  used  for  official  research  purposes 
only.  Responses  to  this  survey  will  be  known  only  to  the  researchers  in 
the  form  of  group  statistics.  Your  tesponses  will  not  be  revealed  to 
anyone  for  any  reason.  This  includes  your  supervisor,  your  co-workers, 
any  subordinates,  your  CPO  personnel  specialists,  and  any  union 
officials.  None  of  these  individuals  will  know  how  you  answered  this 
survey.  Although  respondents  are  identified  by  name  and  SSAN,  the 
research  information  obtained  will  be  used  only  to  improve  evaluation 
and  promotion  techniques  within  the  Air  Force  Civil  Service  Personnel 
System. 

DISCLOSURE  IS  VOLUNTARY:  There  will  be  no  adverse  personnel 
actions  taken  if  you  choose  not  to  participate.  However,  failure  to 
provide  information  would  hinder  the  Air  Force's  ability  to  improve 
the  effectiveness  of  the  civilian  personnel  system.  The  personnel  system 
continues  to  improve  only  with  your  assistance  to  make  additional 
refinements  in  policies  and  procedures.  Your  cooperation  in  this  effort 
is  appreciated.  _ 
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DIRECTIONS 


All  your  marks  should  be  made  on  the  answer  sheet  provided  with  this  survey.  Use  a  soft  lead  pencil 
(Number  1  or  Number  2).  Be  extremely  careful  not  to  make  any  stray  marks  on  the  answer  sheet  because  it 
will  be  machine  scanned,  and  such  marks  will  cause  errors.  If  you  decide  to  change  an  answer,  make  sure 
you  erase  thoroughly.  Your  marks  should  fill  the  answer  oval  and  be  darkened  completely.  Do  not  fold  or 
bend  the  answer  sheet  in  any  way. 

First,  using  the  example  below,  print  the  name  of  the  rated  individual  at  the  bottom  of  the  special 
section  on  the  front  of  the  answer  sheet  called  "RATED  INDIVIDUAL'S  NAME.”  Put  his/her  last  name 
(up  to  12  letters),  then  his/her  first  name  (up  to  7  letters),  and  finally  his/her  middle  initial  in  the  last  slot 
(if  there  is  no  middle  initial,  leave  this  space  blank).  In  the  column  above  each  letter,  blacken  the 
appropriate  oval  for  the  letter  you  have  printed  in  the  box.  Where  you  have  left  a  box  blank,  leave  the 
column  blank. 


Example.  Suppose  the  rated  individual's  name  is  Jonathan  Q.  Doe. 


RATED  INDIVIDUAL'S  NAME 

!  Last  Name 

First  Name 
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Second,  using  the  example  on  the  next  page  write  the  rated  individual's  Social  Security  Account 
Number  (SSAN)  at  the  bottom  of  the  special  section  on  the  front  of  the  answer  sheet  called  "RATED 
INDIVIDUAL'S  SSAN."  (The  rated  individual's  SSAN  is  given  in  paragraph  2  of  the  cover  letter  sent  with 
this  booklet.)  Don't  use  dashes.  Darken  in  the  proper  oval  in  the  column  above  each  number  in  the  SSAN. 


Example.  Suppose  the  rated  individual's  SSAN  is  448-82-7079. 


RATED  INDIVIDUAL'S 
SSAN 
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Third,  using  the  example  below,  write  your  own  Social  Security  Account  Number  (SSAN)  at  the 
bottom  of  the  special  section  on  the  front  of  the  answer  sheet  called  "RATER'S  SSAN."  Don't  use  dashes. 
Darken  in  the  proper  oval  in  the  column  above  each  number  in  the  SSAN. 

Example.  Suppose  your  own  SSAN  is  336-28-4458. 
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Fourth,  turn  to  Section  I  and  code  your  responses  to  the  items  in  the  appropriate  answer  rows  as  in 
this  example: 

Example.  Suppose  you  choose  selection  C  to  item  1,  then  mark  the 
answer  row  1  this  way. 


1  l  a  r;  l<  PT~>C  i  ) 


This  booklet  is  divided  into  10  sections.  Your  answer  sheet  is  also  divided  into  10  sections.  Each 
section  asks  for  a  different  type  of  information.  Make  sure  that  the  section  number  on  the  answer  sheet 
matches  the  section  number  in  this  booklet. 
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Make  sure  that  the  question  number  on  the  answer  sheet  matches  the  question  number  in  this 
booklet. 

This  survey  will  take  you  about  1  hour  to  complete.  This  is  work-associated  and  may  be  completed 
during  normal  duty  hours. 


How  to  Rate 

As  noted  in  the  cover  letter,  this  is  a  study  of  rater  accuracy.  It  is  designed  to  determine  how 
accurately  you  can  rate  other  people,  based  on  what  you  know  about  them.  Therefore,  you  should  try  to 
rate  the  individual  as  accurately  as  you  can. 

Be  objective  with  your  ratings. 

Base  your  judgments  on  what  the  rated  individual  has  done  and  is  doing  now.  Except  for  a  few 
specified  items,  you  should  not  try  to  decide  how  (s)he  will  do  in  the  future. 

As  noted,  the  ratings  will  be  kept  completely  confidential.  The  results  will  not  affect  you  or  the  rated 
individual  in  any  way. 
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SECTION  I:  Job  Performance  Data 


Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  I. 

Blacken  the  letter  on  the  answer  sheet  that  best  answers  each  of  the  following  questions: 

1.  How  would  you  rate  this  individual's  overall  job  performance  in  his/her  present  job? 

A)  Very  poor 

B)  Far  below  average 

C)  Below  average 

D)  Slightly  below  average 

E)  Average 

F)  Slightly  above  average 

G)  Above  average 

H)  Far  above  average 

I)  Outstanding 

2.  Is  this  individual's  present  position  a  supervisory  position  (in  other  words,  is  (s)he  a  rating  official)? 

A)  Yes,  it  is  a  supervisory  position 

B)  No,  it  is  not  a  supervisory  position 

3.  What  do  you  estimate  that  this  individual's  overall  job  performance  would  be  if  (s)he  were  to  be 
promoted  to  the  next  grade  level? 

A)  Very  poor 

B)  Far  below  average 

C)  Below  average 

D)  Slightly  below  average 

E)  Average 

F)  Slightly  above  average 

G)  Above  average 

H)  Far  above  average 

I)  Outstanding 

4.  How  would  you  rate  this  individual's  capability  to  serve  in  a  supervisory  position  (in  other  words,  as  a 
rating  official)? 

A)  Very  poor 

B)  Far  below  average 

C)  Below  average 

D)  Slightly  below  average 

E)  Average 

F)  Slightly  above  average 

G)  Above  average 

H)  Far  above  average 

I)  Outstanding 


5.  How  would  you  rate  this  individual's  capability  to  serve  in  a  managerial  position  (in  other  words, 
making,  interpreting,  and  executing  policy)? 

A)  Very  poor 

B)  Far  below  average 

C)  Below  average 

D)  Slightly  below  average 

E)  Average  • 

F)  Slightly  above  average 

G)  Above  average 

H)  Far  above  average 

I)  Outstanding 

6.  In  your  estimation,  what  pay  schedule  will  this  individual  be  working  under  when  (s)he  retires,  if 
(s)he  stays  with  Federal  service  until  retirement? 

A)  GS 

B)  WG 

C)  WL 

D)  WS 

7.  In  your  estimation,  what  is  the  highest  grade  level  this  individual  will  reach  before  retiring,  if  (s) he 
stays  with  Federal  service  until  retirement?  (assume  that  (s)he  will  be  working  under  the  pay  schedule 
you  marked  in  question  6) 


A) 

1 

G) 

7 

M) 

13 

B) 

2 

H) 

8 

N) 

14 

C) 

3 

1) 

9 

0) 

15 

D) 

4 

J) 

10 

P) 

16  or  above 

E) 

5 

K) 

11 

F) 

6 

L) 

12 
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SECTION  II:  Estimated  Score  on  Vocabulary  Test 


From  your  knowledge  of  the  rated  individual,  you  probably  have  some  knowledge  of  his/her  vocabulary 
skills.  This  would  include  his/her  knowledge  of  different  words,  correct  word  usage,  and  vocabulary  size. 

In  this  section,  you  are  asked  to  estimate  the  rated  individual's  vocabulary  skills. 

Previously,  the  rated  individual  took  a  series  of  tests  that  included  a  vocabulary  test.  The  vocabulary  test 
was  30  questions  long.  For  a  standard  administration  of  the  test  to  Civil  Service  employees,  the  range  of 
scores  generally  runs  from  6  items  correct  to  30  items  correct,  with  an  average  score  of  15  items  correct. 

Blacken  the  circle  in  Section  II  of  the  answer  sheet  that  indicates  the  score  you  think  the  rated  individual 
got  on  the  vocabulary  test.  You  may  assume  that  (s)he  took  the  standard  administration  of  the  test: 

Range:  Bottom  score  =  6  items  correct 
Top  score  =  30  items  correct 

Average  score  =  15  items  correct 
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SECTION  III:  Ratings  of  Trait  Factors 


Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  III. 

In  this  section,  you  will  rate  the  individual  on  12  traits,  according  to  the  following  9-point  scale: 


Very 

Far 

Below 

Slightly 

Average  Slightly 

Above 

Far 

Out- 

Poor 

Below 

Average 

Below 

Above 

Average 

Above 

Standing 

Average 

Average 

Average 

Average 

The  list  of  12  traits  you  will  use  for  rating  is  shown  below. 

You  should  blacken  the  corresponding  numbers  (1-9)  in  items  A  through  L  on  Section  ill  of  the  answer 
sheet,  depending  on  how  you  rate  the  individual  on  each  trait. 

For  example,  if  you  feel  that  (s)he  rates  "Above  Average"  on  "Energy,"  you  would  blacken  circle  "7"  for 
item  "A."  If  you  feel  that  (s)he  rates  "Far  Below  Average"  on  "Skill,"  you  would  blacken  circle  "2"  for 
item  "B." 

Rate  the  individual  on  every  trait,  even  if  you  know  litt'a  or  nothing  about  his/her  strength  or  weakness  on 
that  trait. 

Please  read  the  list  of  traits  and  their  definitions  carefully.  Then  fill  in  the  appropriate  ratings  on  the  answer 
sheet. 


TRAIT  FACTORS 


A.  Energy:  Shows  vigor  on  the  job;  seems  to  prefer  a  fast  pace;  willing  to  exert  effort  in  accomplishing 

tasks. 

B.  Skill:  Performs  job-associated  tasks  well,  whether  they  require  physical,  mechanical,  technical, 

professional,  or  managerial  skills;  considered  expert  by  co-workers. 

C.  Completion:  Follows  through  well;  accomplishes  all  tasks  required  to  complete  a  job  in  a  timely 

manner  on  his/her  own;  hates  to  leave  a  task  unfinished. 

D.  Self-sufficiency:  Works  independently  with  little  need  for  supervision  or  help  from  co-workers. 

E.  Leadership:  Inspires  others  to  action;  accomplishes  goals  by  organizing  others  and  influencing  their 

behavior. 

F.  Responsiveness:  Follows  instructions;  understands  and  carries  out  oral  and  written  instructions  to  the 

best  of  his/her  ability. 

G.  Understanding  of  Others:  Understands  the  behaviors  of  fellow  workers,  superiors,  and  subordinates; 

can  "stand  in  the  other  person's  shoes." 

H.  Originality:  Devises  new  solutions  to  problems;  creates  new  methods  and  procedures  for  accomplishing 

objectives. 

I.  Pride  in  Work:  Enjoys  doing  good  work  and  producing  a  first-rate  product;  likes  to  be  the  best  at 

whatever  <s)he  does. 

J.  Learning  Ability:  Picks  up  new  ideas  and  procedures  very  quickly;  easy  to  train;  can  change  to  meet  the 

demands  of  the  situation. 

K.  Speaking  Ability:  Explains,  instructs,  and  converses  with  others  in  a  clear  and  effective  manner. 

L.  Writing  Ability:  Prepares  memos,  letters,  reports,  instructions,  and  other  written  materials  that  are 

effective  and  dearly  understood. 


SECTION  IV:  Ratings  of  General  Abilities 
Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  IV. 


In  this  section,  you  will  rate  the  individual  on  a  set  of  abilities.  These  abilities  are  similar  to  the  traits  used 
in  the  previous  section.  However,  they  describe  some  different  characteristics.  Use  the  9-point  scale  below. 


A 


Very 

Poor 


2 

Far 

Below 

Average 


3 

Below 

Average 


Slightly 

Below 

Average 


5 

Average 


Slightly 

Above- 

Average 


7 

Above 

Average 


8 

Far 

Above 

Average 


9 

Out¬ 

standing 


The  list  of  eight  abilities  you  will  use  for  rating  is  shown  below. 


Rate  the  individual  on  every  ability,  even  if  you  know  little  or  nothing  about  his/her  strength  or  weakness 
on  that  ability. 


Please  read  the  list  of  abilities  and  their  definitions  carefully.  Then  fill  in  the  appropriate  ratings  on  the 
answer  sheet. 


GENERAL  ABILITIES 


A.  Arithmetic  Computation:  Good  at  adding,  subtracting,  multiplying,  and  dividing  numbers;  manipulates 

numbers  rapidly  and  accurately. 

B.  Word  Knowledge:  Uses  many  different  words  in  conversation  and  writing;  large  vocabulary;  uses  words 

correctly. 

C.  Data  Interpretation:  Good  at  drawing  conclusions  and  making  interpretations  from  data  presented  in 

the  forms  of  graphs,  charts,  and  tables. 

D.  Electrical  Information:  Good  at  solving  electrical  problems  by  applying  knowledge  of  electricity  and 

electronics. 


E.  Shop  Procedures:  Has  a  wide  knowledge  of  tools;  good  at  selecting  the  proper  tool  for  use  in  a  specific 
situation  or  for  a  specific  task. 


F.  Mechanical  Skill:  Good  at  troubleshooting  and  repairing  various  mechanical  devices;  can  understand  the 

operating  characteristics  of  mechanical  devices  by  examining  and  studying  their  parts. 

G.  Carefulness:  Takes  the  necessary  time  and  effort  to  produce  work  which  has  few  errors;  hates  to  make 

mistakes;  would  rather  work  slowly  than  take  a  chance  on  making  errors. 


H.  Reading  Ability:  Correctly  understands  written  instructions;  knows  the  meaning  of  what  he/she  reads. 
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SECTION  V:  Rankings  of  Trait  Factors 


Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  V. 

In  this  section,  you  will  rank  the  traits  you  used  in  Section  III,  according  to  how  well  they  apply  to  the 
rated  individual. 

For  instance,  if  you  are  ranking  the  traits  for  an  individual,  Jonathan  Doe,  and  you  decide  that  his  most 
outstanding  trait  is  "Originality,"  you  would  give  that  trait  a  rank  of  1.  It  doesn't  matter  if  every  other 
person  in  the  organization  has  more  originality  than  Jonathan.  If  that  is  Jonathan's  best  trait,  give  it  a  rank 
of  1. 

Use  each  number  only  once.  Give  the  individual's  most  outstanding  trait  a  1.  Give  his/her  next  most 
outstanding  trait  a  2.  Continue  this  way  until  you  have  only  one  trait  left— his/her  least  outstanding 
trait-which  you  give  a  rank  of  12. 

Rank  all  12  traits  even  if  you  know  little  or  nothing  about  the  individual's  strengths  or  weaknesses  on  some 
traits. 

The  completed  section  of  a  sample  answer  sheet  below  shows  the  ranking  of  Jonathan  Doe.  You  can  see 
that  the  rater  ranked  "Originality"  as  number  1  for  Jonathan.  Actually,  there  are  a  number  of  people  in  the 
organization  with  more  originality  than  Jonathan.  But,  originality  is  still  Jonathan's  best  trait,  so  it  must  be 
given  a  1. 

Note  that  each  trait  was  given  a  different  ranking.  Make  sure  that  you  blacken  only  one  number  for  each 
trait.  Also,  make  sure  that  you  blacken  a  number  for  every  trait,  even  if  you  are  unsure  of  some  of  your 
rankings. 


Sample  Answer  Sheet 


SECTION  V 

Rankings  of  Trait  Factors 

A  CDC»C33*CX)GOCDCjDCa3CIS)aX)CI5) 

8  CDC2DCXDCID«»CDCiDCSDC3DCia)CnDCIZ) 
C  CDCSC5DGDCDCiDCzDCDCDC3S>CIP*» 
D  CDf»C33CI3C30CDCxDCOCaDCIcDQTTCID 
E  CDC3D*B*CDC5DC£DCaDCaDC5DCia)aDCj2) 
F  ODC5DCOGDC^GDCOCaDC5DCiS)«B»CI2) 

G  CIDCZDGDC50CEDC3DW*COCDCI1DCIDCIJD 
H  «»CZOC^CiDC5DCDCXjCOCSDaiDaaDCED 
I  CDC^CDC3DQDCDCD«»C^CI©<XDaZ) 
J  CIDCZDCDCCC!D«»CZDCOCSDaB)CODCCD 
K  CDCZDCZ3DC3DC3DC2DC2DCjDC53*B»aX)CI2? 
L  CIDC3DC3DC30C5I)CEDCZDCBDSBPCIZDCCDaCD 


The  list  of  traits  is  shown  on  the  next  page.  Please  review  the  traits  and  their  definitions.  Then  fill  in  the 
appropriate  numbers  on  your  answer  sheet. 


TRAIT  FACTORS 


A.  Energy:  Shows  vigor  on  the  job;  seems  to  prefer  a  fast  pace;  willing  to  exert  effort  in  accomplishing 

tasks. 

B.  Skill:  Performs  job-associated  tasks  well,  whether  they  require  physical,  mechanical,  technical, 

professional,  or  managerial  skills;  considered  expert  by  co-workers. 

C.  Completion:  Follows  through  well;  accomplishes  alt  tasks  required  to  complete  a  job  in  a  timely 

manner  on  his/her  own;  hates  to  leave  a  task  unfinished. 

D.  Self-sufficiency:  Works  independently  with  little  need  for  supervision  or  help  from  co-workers. 

E.  Leadership:  Inspires  others  to  action;  accomplishes  goals  by  organizing  others  and  influencing  their 

behavior. 

F.  Responsiveness:  Follows  instructions;  understands  and  carries  out  oral  and  written  instructions  to  the 

best  of  his/her  ability. 

G.  Understanding  of  Others:  Understands  the  behaviors  of  fellow  workers,  superiors,  and  subordinates; 

can  “stand  in  the  other  person's  shoes." 

H.  Originality:  Devises  new  solutions  to  problems;  creates  new  methods  and  procedures  for  accomplishing 

objectives. 

I.  Pride  in  Work:  Enjoys  doing  good  work  and  producing  a  first-rate  product;  likes  to  be  the  best  at 

whatever  (s)he  does. 

J.  Learning  Ability:  Picks  up  new  ideas  and  procedures  very  quickly; easy  to  train;  can  change  to  meet  the 

demands  of  the  situation. 

K.  Speaking  Ability:  Explains,  instructs,  and  converses  with  others  in  a  clear  and  effective  manner. 

L.  Writing  Ability:  Prepares  memos,  letters,  reports,  instructions,  and  other  written  materials  that  are 

effective  and  clearly  understood. 
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SECTION  VI:  Rankings  of  General  Abilities 

Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  VI. 

In  this  section,  you  will  rank  the  general  abilities  you  used  in  Section  IV,  according  to  how  well  they  apply 
to  the  rated  individual. 

As  in  the  previous  section,  you  are  comparing  the  degree  to  which  each  ability  applies  to  the  rated  person 
only.  You  must  use  each  number  only  once. 

Rank  all  eight  abilities,  even  if  you  know  little  or  nothing  about  the  individual's  strengths  or  weaknesses  on 
some  abilities. 

Make  sure  that  you  blacken  only  one  number  for  each  ability.  Also,  make  sure  that  you  blacken  a  number 
for  every  ability,  even  if  you  are  unsure  of  your  rankings. 

The  list  of  abilities  is  shown  below.  Please  review  the  abilities  and  their  definitions.  Then  fill  in  the 
appropriate  numbers  on  your  answer  sheet. 


GENERAL  ABILITIES 

A.  Arithmetic  Computation:  Good  at  adding,  subtracting,  multiplying,  and  dividing  numbers;  manipulates 

numbers  rapidly  and  accurately. 

B.  Word  Knowledge:  Uses  many  different  words  in  conversation  and  writing;  large  vocabulary;  uses  words 

correctly. 

C.  Data  Interpretation:  Good  at  drawing  conclusions  and  making  interpretations  from  data  presented  in 

the  forms  of  graphs,  charts,  and  tables. 

D.  Electrical  Information:  Good  at  solving  electrical  problems  by  applying  knowledge  of  electricity  and 

electronics. 

E.  Shop  Procedures:  Has  a  wide  knowledge  of  tools;  good  at  selecting  the  proper  tool  for  use  in  a  specific 

situation  or  for  a  specific  task. 

F.  Mechanical  Skill:  Good  at  troubleshooting  and  repairing  various  mechanical  devices;  can  understand  the 

operating  characteristics  of  mechanical  devices  by  examining  and  studying  their  parts. 

G.  Carefulness:  Takes  the  necessary  time  and  effort  to  produce  work  which  has  few  errors;  hates  to  make 

mistakes;  would  rather  work  slowly  than  take  a  chance  on  making  errors. 

H.  Reading  Ability:  Correctly  understands  written  instructions;  knows  the  meaning  of  what  he/she  reads. 
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SECTION  VII:  Additional  Information 

Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  VII. 

Compared  to  other  individuals  doing  about  the  same  type  of  work,  does  the  rated  individual: 

1.  Complete  his/her  work  (projects,  duties,  and  tasks)  faster? 

A)  Much  faster 

B)  Somewhat  faster 

C)  About  average 

D)  Somewhat  slower 

E)  Much  slower 

2.  Work  more  productively  during  his/her  work  time? 

A)  Much  more  productively 

B)  Somewhat  more  productively 

C)  About  average 

D)  Somewhat  less  productively 

E)  Much  less  productively 

3.  Spend  a  greater  percentage  of  time  on  the  job  working,  as  opposed  to  socializing,  sitting  idle,  involved 
in  personal  affairs,  taking  breaks,  and  so  forth? 

A)  Much  more  time  working 

B)  Somewhat  more  time  working 

C)  About  average 

D)  Somewhat  less  time  working 

E)  Much  less  time  working 

✓  _ 

4.  Show  more  initiative  in  starting,  carrying  out,  and  completing  projects? 

A)  Much  more  initiative 

B)  Somewhat  more  initiative 

C)  About  average 

D)  Somewhat  less  initiative 

E)  Much  less  initiative 

Most  promotions  involve  taking  on  additional  duties,  responsibilities,  training,  effort,  etc.,  for  successful  job 
performance.  Which  of  the  following  responsibilities  or  activities  do  you  believe  the  rated  individual  would 
accept  for  promotion  to  the  next  grade  level? 

5.  Enter  a  training/retraining  program  on  government  time  and  expense? 

A)  Yes 

B)  No 

6.  Enter  a  training/retraining  program  on  his/her  own  time  but  at  government  expense? 

A)  Yes 

B)  No 
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7.  Enter  a  training/retraining  program  on  his/her  own  time  and  own  expense? 


A)  Yes 

B)  No 

8.  Move  to  another  brandi/section  within  the  present  organization? 

A)  Yes 

B)  No 

9.  Move  to  another  organization  on  base? 

A)  Yes 

B)  No 

10.  Move  to  another  base  within  the  same  local  area? 

A)  Yes 

B)  No 

1 1 .  Move  to  another  base  outside  the  local  area? 

A)  Yes 

B)  No 

1 2.  Accept  a  job  outside  his/her  current  job  series? 

A)  Yes 

B)  No 

13.  Devote  more  time  to  administrative/manageriai  duties? 

A)  Yes 

B)  No 

14.  Do  more  work  (increase  production)? 

A)  Yes 

B)  No 

1 5.  Increase  the  quality  of  his/her  work? 

A)  Yes 

B)  No 

16.  Improve  his/her  working  relationships  with  supervisor,  co-workers,  or  subordinates? 

A)  Yes 

B)  No 
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17.  Assume  extra  additional  duties? 


A)  Yes 

B)  No 

18.  Take  more  initiative  on  important  projects? 

A)  Yes 

B)  No 

1 9.  Perform  additional  supervisory  tasks? 

A)  Yes 

B)  No 

20.  Assume  high  pressure  tasks  or  duties,  such  as  public  briefings,  working  on  tight  deadlines, 
negotiations,  or  dealing  with  other  agencies  on  major  projects? 

A)  Yes 

B)  No 

21.  Workovertime? 

A)  Yes 

B)  No 
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SECTION  VIII:  Rankings  of  Job  Characteristics  on  Trait  Factors 

Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  VIII. 

In  this  section,  you  will  be  ranking  the  rated  individual's  job,  instead  of  the  individual,  himself. 

The  same  12  traits  and  definitions  you  used  in  Section  III  are  listed  below.  This  time  you  will  rank  them  to 
show  how  important  they  are  to  successful  performance  of  the  individual's  job. 

For  example,  if  you  think  "Leadership"  is  the  most  important  trait  required  for  doing  well  in  this  job,  you 
would  give  that  trait  a  rank  of  1.  The  next  most  important  trait  would  get  a  rank  of  2.  Continue  this  way 
until  you  have  only  one  trait  left-the  one  least  important  for  die  job— which  you  give  a  rank  of  12. 

In  this  section,  you  are  not  ranking  the  individual.  Instead,  you  are  ranking  the  job  requirements.  Do  not 
consider  how  well  or  how  poorly  the  individual  fulfills  these  requirements.  Instead,  rank  each  trait  on  how 
important  it  is  to  successful  performance  of  the  job. 

Make  sure  that  you  blacken  only  one  number  for  each  trait.  Also,  make  sure  that  you  blacken  a  number  for 
every  trait,  even  if  you  are  unsure  of  your  rankings. 

TRAIT  FACTORS 

A.  Energy:  Shows  vigor  on  the  job;  seems  to  prefer  a  fast  pace;  willing  to  exert  effort  in  accomplishing 

tasks. 

B.  Skill:  Performs  job-associated  tasks  well,  whether  they  require  physical,  mechanical,  technical, 

professional,  or  managerial  skills;  considered  expert  by  co-workers. 

C.  Completion:  Follows  through  well;  accomplishes  all  tasks  required  to  complete  a  job  in  a  timely 

manr<er  on  his/her  own;  hates  to  leave  a  task  unfinished. 

D.  Self-sufficiency:  Works  independently  with  little  need  for  supervision  or  help  from  co-workers. 

E.  Leadership:  Inspires  others  to  action;  accomplishes  goals  by  organizing  others  and  influencing  their 

behavior. 

F.  Responsiveness:  Follows  instructions;  understands  and  carries  out  oral  and  written  instructions  to  the 

best  of  his/her  ability. 

G.  Understanding  of  Others:  Understands  the  behaviors  of  fellow  workers,  superiors,  and  subordinates; 

can  "stand  in  the  other  person's  shoes." 

H.  Originality:  Devises  new  solutions  to  problems;  creates  new  methods  and  procedures  for  accomplishing 

objectives. 

I.  Pride  in  Work:  Enjoys  doing  good  work  and  producing  a  first-rate  product;  likes  to  be  the  best  at 

whatever  (s)he  does. 

J.  Learning  Ability:  Picks  up  new  ideas  and  procedures  very  quickly;  easy  to  train;  can  change  to  meet  the 

demands  of  the  situation. 

K.  Speaking  Ability:  Explains,  instructs,  and  converses  with  others  in  a  clear  and  effective  manner. 

L.  Writing  Ability:  Prepares  memos,  letters,  reports,  instructions,  and  other  written  materials  that  are 

effective  and  clearly  understood. 
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SECTION  IX:  Rankings  of  Job  Characteristics  on  General  Abilities 


Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  IX. 

Below  is  a  list  of  the  same  eight  general  abilities  and  definitions  you  used  in  Section  IV. 

This  time  you  will  rank  them  to  show  how  important  they  are  to  successful  performance  of  the  individual's 
job,  just  as  you  did  with  the  trait  factors  in  the  previous  section. 

As  in  the  previous  section,  you  are  not  ranking  the  individual.  Instead,  you  are  ranking  the  job 
requirements.  Do  not  consider  how  well  or  how  poorly  the  individual  fulfills  these  requirements.  Instead, 
rank  each  ability  on  how  important  it  is  to  successful  performance  of  the  job. 

Make  sure  that  you  blacken  only  one  number  for  each  ability.  Also,  make  sure  that  you  blacken  a  number 
for  every  ability,  even  if  you  are  unsure  of  your  rankings. 


GENERAL  ABILITIES 

A.  Arithmetic  Computation:  Good  at  adding,  subtracting,  multiplying,  and  dividing  numbers;  manipulates 

numbers  rapidly  and  accurately. 

B.  Word  Knowledge:  Uses  many  different  words  in  conversation  and  writing;  large  vocabulary;  uses  words 

correctly. 

C.  Data  Interpretation:  Good  at  drawing  conclusions  and  making  interpretations  from  data  presented  in 

the  forms  of  graphs,  charts,  and  tables. 

D.  Electrical  Information:  Good  at  solving  electrical  problems  by  applying  knowledge  of  electricity  and 

electronics. 

E.  Shop  Procedures:  Has  a  wide  knowledge  of  tools;  good  at  selecting  the  proper  tool  for  use  in  a  specific 

situation  or  for  a  specific  task. 

F.  Mechanical  Skill:  Good  at  troubleshooting  and  repairing  various  mechanical  devices;  can  understand  the 

operating  characteristics  of  mechanical  devices  by  examining  and  studying  their  parts. 

G.  Carefulness:  Takes  the  necessary  time  and  effort  to  produce  work  which  has  few  errors;  hates  to  make 

mistakes;  would  rather  work  slowly  than  take  a  chance  on  making  errors. 

H.  Reading  Ability.  Correctly  understands  written  instructions;  knows  the  meaning  of  what  he/she  reads. 
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SECTION  X:  Background  Data 


Locate  the  block  of  answer  spaces  on  your  answer  sheet  identified  as  SECTION  X. 

The  previous  sections  dealt  with  the  rated  individual  and  his/her  job. 

The  following  items  concern  you  and  your  job.  This  information  will  provide  a  better  understanding  of  how 
people  do  ratings. 

Please  answer  the  following  questions  about  yourself. 

1 .  What  is  your  relationship  to  the  rated  individual? 

A)  Co-worker  of  the  rated  individual 

B)  Supervisor  (Rating  Official)  of  the  rated  individual 

C)  Associate  of  the  Supervisor  of  the  rated  individual 

2.  What  is  your  military /civilian  status? 

A)  Military 

B)  Civilian 

3.  What  is  your  pay  schedule? 

A)  GS 

B)  WG 

C)  WL 

D)  WS 

E)  Military  . 

i 

j 

4.  What  is  your  current  civilian  grade  level?  ' 

A)  Not  applicable  (Military) 

B)  1,2,  or  3 

C)  4,  5,  or  6 

D)  7,  8,  or  9 

E)  10,  11,  or  12 

F)  13, 14,  or  15 

G)  16  or  above 

5.  What  is  your  current  military  grade  level? 

A)  Not  applicable  (Civilian) 

B)  Airman  Basic,  Airman,  Airman  First  Class,  or  Senior  Airman 

C)  Sergeant,  Staff  Sergeant,  or  Technical  Sergeant 

D)  Master  Sergeant,  Senior  Master  Sergeant,  or  Chief  Master  Sergeant 

E)  Second  Lieutenant,  First  Lieutenant,  or  Captain 

F)  Major,  Lieutenant  Colonel,  or  Colonel 

G)  Brigadier  General  or  above 


6.  What  is  the  highest  education  level  you  have  reached? 


A)  Eighth  grade  or  lower 

B)  Attended  high  school;  did  not  graduate 

C)  Completed  high  school  with  graduation 

D)  Attended  two  years  or  less  college 

E)  Associate  degree  or  more  than  two  years  of  college 

F)  Bachelor's  degree 

G)  Bachelor's  degree  with  some  graduate  work;  no  graduate  degree 

H)  Master's  degree  or  PhD 

7.  How  many  years  of  total  active  Federal  service  do  you  have? 

A)  Fewer  than  5  years 

B)  5-9  years 

C)  1 0-14  years 

D)  15-1 9  years 

E)  20  or  more  years 

8.  How  many  years  have  you  spent  in  your  current  job  series  (civilian)  or  career  field 
(military)? 

A)  Fewer  than  5  years 

B)  5-9  years 

C)  10-14  years 

D)  15-1 9  years 

E)  20  or  more  years 

9.  How  many  years  have  you  spent  in  your  current  job? 


A)  Less  than  1  year 

B)  1—3  years 

C)  4-6  years 

D)  7-9  years 

E)  1 0  or  more  years 


10.  How  many  years  have  you  spent  in  a  supervisory  position  of  any  sort? 

A)  Not  applicable;  I  have  never  been  a  supervisor 

B)  Fewer  than  5  years 

C)  5-9  years 

D)  10-14  years 

E)  15-1 9  years 

F)  20  or  more  years 

1 1 .  How  many  civilian  workers  do  you  directly  supervise  (as  a  rating  official? 

A)  None 

B)  1-2 

C)  3-4 

D)  5-6 

E)  7-8 

F)  9  or  more 
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12.  How  many  total  workers  (both  military  and  civilian)  do  you  directly  supervise  (as  a  rating  official)? 

A)  None 

B)  1-2 

C)  3-4 

D)  5-6 

E)  7-8 

F)  9  or  more 

13.  How  accurately  do  you  feel  you  can  rate  other  people,  in  general? 

A)  Extremely  accurately 

B)  Very  accurately 

C)  Somewhat  accurately 

D)  Not  very  accurately 

E)  Not  at  all  accurately  t , 

14.  How  familiar  are  you  with  the  rated  individual  and  his  job  performance? 

A)  Minimally  (I  know  the  individual  but  I  am  not  familiar  with  his/her  job  performance) 

B)  Not  very  familiar 

C)  Moderately  familiar 

D)  Very  familiar 

E)  Extremely  familiar 

1 5.  How  long  have  you  known  the  rated  individual? 

A)  Less  than  6  months 

B)  6  months  -  1 1  months 

C)  1 2  months  -  23  months 

D)  2-5  years 

E)  More  than  5  years 

16.  How  long  have  you  supervised  (as  a  rating  official)  the  rated  individual? 

A)  Not  applicable  (for  co-workers  of  the  rated  individual  and  associates  of  supervisors) 

B)  Less  than  6  months 

C)  6  months  —  1 1  months 

D)  12  months  —  23  months 

E)  2—5  years 

F)  More  than  5  years 

17.  On  the  average,  how  often  do  you  come  in  contact  with  the  rated  individual? 

A)  More  than  once  a  day 

B)  Once  a  day 

C)  More  than  once  a  week  but  less  than  once  a  day 

D)  Once  a  week 

E)  Less  than  once  a  week 
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18.  Do  you  know  the  rated  individual  more  by  direct  observation  or  by  his/her  reputation? 

A)  Entirely  by  direct  observation 

B)  Mostly  by  direct  observation 

C)  Equally  by  direct  observation  and  reputation 

D)  Mostly  by  reputation 

E)  Entirely  by  reputation 

19.  How  accurate  do  you  think  your  judgments  of  the  rated  individual  were? 

A)  Extremely  accurate 

B)  Very  accurate 

C)  Moderately  accurate 

D)  Very  inaccurate 

E)  Extremely  inaccurate 

20.  The  Department  of  the  Air  Force  Civilian  Personnel  Office  is  considering  the  use  of  a  promotion 
system  that  resembles  the  one  you  used  in  this  project.  The  proposed  system  would  include  the 
Civilian  Personnel  Examination,  experience/background  information,  and  ratings  by  people  who 
know  you.  How  do  you  think  this  system  compares  with  the  current  system  for  making  promotions? 

A)  The  proposed  system  is  much  better  than  the  current  system 

B)  The  proposed  system  is  somewhat  better  than  the  current  system 

C)  The  two  systems  are  about  the  same 

D)  The  proposed  system  is  somewhat  worse  than  the  current  system 

E)  The  proposed  system  is  much  worse  than  the  current  system 


When  you  have  answered  all  the  questions,  please  do  the  following: 

1 .  Check  over  your  answer  sheet: 

•  Make  sure  that  the  rated  individual's  name  and  SSAN  and  your  SSAN  are  coded  properly 

•  Make  sure  that  you  have  answered  every  item 

•  Make  sure  that  there  are  no  stray  marks  or  incomplete  erasures 

•  Make  sure  that  you  have  used  each  rank  once  and  only  once  in  Sections  V,  VI,  VIII,  and  IX 
(ranking  sections). 

2.  Put  the  following  items  in  the  return  envelope: 

Job/Worker  Characteristics  Answer  Sheet 
Job/Worker  Characteristics  Ratings  Booklet  (this  booklet) 

t  * 

Make  sure  that  you  do  not  fold  or  wrinkle  the  answer  sheet.  Seal  the  envelope  and  return  it  to  the  Civilian 
Personnel  Office. 
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APPENDIX  C:  PROMOTION  POLICY  DATA  PANEL  PROCEDURES 
AND  PROMOTION  WEIGHTING  SYSTEM  DEVELOPMENT 

I.  OVERVIEW 

The  CONUS-wlde  field  test  provided  12,856  usable  data  sets  comprised  of 
demographic  Information  and  rating  data.  As  discussed  In  Appendix  B,  these 
two  types  of  data  had  been  Identified  as  appropriate  for  use  In  an 
operational  appraisal  system  (from  a  regulatory,  legal,  and  psychometric 
standpoint).  However,  before  the  data  elements  could  be  presented  to  a 
panel  for  the  development  of  an  appropriate  weighting  system  of  desired 
promotion  factors,  the  large  number  of  candidate  promotion  system  factors 
available  from  the  data  sets  had  to  be  reduced  considerably.  The  210  data 
elements  In  the  Demographic  Questionnaire-Worker  (102  Items)  and  Job/Worker 
Characteristics  Ratings  (108  Items)  booklets  presented  a  formidable  array  of 
candidate  variables  for  use  In  the  operational  CPAS.  Although  Promotion 
Policy  Data  Panels  of  experienced  personnel  were  to  be  used  to  Identify 
variables  specifically  related  to  performance  In  their  job  family,  they 
could  not  be  expected  to  handle  such  a  large  pool  of  Items.  Since  these 
panel  members  could  not  be  expected  to  evaluate  the  Items  on  their 
psychometric  characteristics  and  legal  Implications,  a  panel  composed  of 
Industrial  psychologists  under  contract  and  from  AFHRL  and  personnel 
specialists  from  OCPO  was  tasked  to  reduce  the  pool  of  variables.  These 
panel  members  were  knowledgeable  In  the  personnel  system,  applicable  case 
law,  and/or  psychometrics. 

Since  a  high  proportion  of  selection  systems  have  been  determined  to 
create  adverse  Impact  on  minority  groups  (United  States  Office  of  Personnel 
Management,  1979;  Casclo  &  Bernardln,  1981),  particular  attention  was  paid 
to  the  requirements  of  the  Uniform  Guidelines  for  Employee  Selection  during 
the  variable  reduction  phase.  The  guidelines  say  In  effect  that  a  selection 
system  which  creates  adverse  Impact  1$  defensible  If  the  development  process 
tried  to  Identify  all  sources  of  Information  which  could  be  related  to  job 
performance  and  then  selected  for  operational  use  those  Information  Items 
which  minimized  adverse  Impact;  but  only  If  the  selected  set  of  Items 
provides  a  highly  valid  selection  system.  In  other  words,  try  to  select 
Items  that  minimize  adverse  Impact  but  do  not  sacrifice  system  validity  In 
the  process. 

To  reduce  the  210  demographic  and  rating  variables  to  a  workable  subset, 
a  large  data  matrix  was  constructed  for  rating  each  of  the  elements  on  a 
series  of  selection  criteria.  Four  basic  types  of  criteria  were  used:  (a) 
estimated  validity,  (b)  uniqueness,  (c)  bias,  and  (d)  legal  defenslblllty. 
Correlations  of  the  variables  with  peer  ratings  of  overall  performance  and 
projected  performance  at  the  next  higher  levels  served  as  estimates  of  Item 
validity.  A  computing  process  titled  VARSEL  (Gould  &  Christa!,  1976),  was 
used  to  develop  Indices  of  Item  uniqueness  and  redundancy  from  an  Item 
Inter  correlation  matrix.  Comparisons  of  mean  ratings  by  race  and  sex 
category  were  used  to  evaluate  Item  bias.  Thus  objective  measures  of  each 
Item's  validity,  uniqueness,  and  bias  were  available.  Subjective  ratings  of 
legal  defenslblllty  were  made  by  personnel  specialists  and  psychologists 
based  on  case  law. 
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After  the  objective  Measures  were  available,  a  panel  of  eight  research 
psychologists  and  two  personnel  specialists  was  convened  to  select  a 
workable  subset  of  variables  froM  the  data  Matrix,  using  the  criterion 
Indices  for  guidance.  All  data  IteMS  were  rated  on  a  contlnuuM  of  their 
legal  defenslblllty  using  the  guidelines  Interpretable  froM  case  law  on 
equity  and  fairness  In  selection  procedures,  and  those  with  low  ratings  were 
deleted.  This  procedure  significantly  reduced  the  pool  of  variables.  This 
first  procedure  reduced  the  pool  of  variables  to  observable  behaviors  and 
deleted  demographic  variables  Most  related  to  past  opportunities  and  thus 
associated  with  past  dlscrlMlnatlons.  A  consensus  of  panel  judgMents  then 
selected  the  final  subset  of  variables  consisting  of  24  rating  eleMents: 
three  overall  ratings  (job  performance,  supervisory.  Managerial);  12 
behavioral  ratings  such  as  responsiveness  to  directions,  self-sufficiency, 
etc;  three  ability  ratings  (quantitative,  reading,  data  Interpretation); 
four  Motivation  Indicators  (productivity.  Initiative,  speed  of  coMpletlon, 
and  aMOunt  of  working  tlMe  spent  In  productive  efforts);  and  two  cowposlte 
variables  which  cowblned  behavioral  and  Motivational  Measures.  The  specific 
eleMents  chosen  were  all  contained  In  the  Job/Worker  Characteristics  Ratings 
booklet:  Section  I  IteMS  1,  4,  and  5;  Section  III  Itews  A  through  L; 
Sections  IV  IteMS  A,  C,  and  H;  and  Section  VII  IteMS  1  through  4  (see 
Appendix  B-2  for  detailed  IteM  descriptions).  Once  the  final  subset  of  24 
rating  eleMents  was  selected,  the  next  task  was  to  Identify  the  relative 
relationship  of  each  element  to  job  perforMance  In  each  job  faMlly  and 
specify  how  those  ratings  were  to  be  used  In  the  operational  CPAS.  This 
appendix  outlines  In  considerable  detail  the  process  by  which  candidate 
proMotlon  factors  applicable  for  each  of  the  job  faMllles  were  Identified 
and  appropriate  weights  developed  for  each  of  the  factors.  Included  are  the 
panel  MeMber  selection  criteria  and  procedures,  the  list  of  candidate 
factors,  subject  saMple  and  data  preparation,  adMlnlstratlve  procedures  used 
for  the  panels,  the  results  of  the  analysis  of  the  panel  data,  and  the 
resulting  proMotlon  weighting  systews.  Twenty-three  panels,  one  for  each 
civilian  job  faMlly,  were  asseMbled  to  provide  a  content-valid  proMotlon 
systeM  that  would  consist  of  a  restricted  set  of  6  to  10  weighted  prowotlon 
factors.  Each  panel  perfonied  the  two  basic  tasks  of  first  selecting  6  to 
10  proMotlon  factors  froM  a  list  of  24  candidate  factors  and  then 
rank-ordering  (sorting)  a  selected  randoM  saMple  of  eMployees  In  terns  of 
the  eMployees'  proMotablllty  as  reflected  by  these  factors.  Inconsistent 
raters  were  Identified  and  rewoved  frow  further  analyses.  Judgment  analysis 
techniques  were  used  to  capture  Mathewatlcally  the  weights  that  panel 
Mewbers  applied  to  the  proMotlon  factors  In  rank-ordering  eMployees.  The 
resulting  proMotlon  factors  and  weights  Identified  In  these  analyses  are 
unique  cowblnatlons  for  each  of  the  23  job  faMllles  by  supervisory  or 
nonsupervlsory  category.  The  Directorate  of  Civilian  Personnel  has  declared 
that  the  actual  proMotlon  weights  are  "Privileged  Information"  and  cannot  be 
Included  In  the  body  of  this  paper. 

II.  INTRODUCTION 

Hajor  Objectives 

The  prlwary  purpose  of  the  ProMotlon  Policy  Data  Panel  (PPDP)  exercise 
was  to  capture  the  promotion  policy  of  a  group  of  experienced  Federal 
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Service  career-field  representatives,  so  that  their  policy  (when  converted 
to  a  mathematical  algorithm)  wight  be  specified  and  used  In  conjunction  with 
the  Promotions  and  Placements  Referral  System  (PPRS).  The  PPRS  Is  a 
centralized  computer  system  that  contains  the  current  background, 
experience,  and  job  performance  ratings  on  all  appropriated  fund  Air  Force 
civilian  employees  and  Is  used  to  Identify  and  rank  all  ellglbles  for  each 
vacancy.  It  was  decided  that  CPAS  would  be  used  as  the  last  ranking  factor 
when  the  number  of  ellglbles  exceeded  the  feasible  number  that  could  be 
referred  for  a  vacancy  consideration.  For  this  type  of  application.  It  was 
mandatory  that  a  common  set  of  rating  factors  for  all  ellglbles  must  be 
identified  even  If  not  all  factors  are  used  for  all  jobs.  Given  this 
purpose,  there  were  three  specific  objectives  of  the  panel  exercise.  The 
first  objective  was  to  develop  restricted  lists  of  rating  elements  that 
would  be  maximally  relevant  to  each  of  the  23  civilian  job  families.  The 
second  objective  was  to  obtain  a  job-family-specific  weight  for  each  factor, 
so  that  the  factors  might  be  combined  Into  a  single  promotion  system,  or 
"promotablllty  score,"  for  each  job  family.  The  third  objective  was  to 
obtain  Interim  evidence  for  the  content  validity  of  this  ranking  system, 
until  a  full-scale,  criterion-related  validity  study  could  be  completed 
after  the  CPAS  became  operational. 

After  selection  of  the  workable  subset  of  variables  from  the  data 
collection  Instruments,  some  methodology  had  to  be  used  to  Identify  and 
weight  the  specific  promotion  factors  (l.e.,  variables).  Several  techniques 
were  considered  for  possible  use.  However,  In  the  absence  of  an  external 
criterion  of  promotablllty,  the  traditional  multiple  regression-based 
technique  was  not  considered  appropriate  for  the  selection  of  the  final 
factors  and  their  weights.  Another  possibility  considered  was  the  use  of  a 
panel  of  management/personnel  experts  who  could  Identify  the  final  list  of 
factors;  however,  this  technique  would  have  given  no  assurances  about  the 
content  relevance  or  validity  of  the  final  factors.  The  use  of  an 
equal -weighting  scheme,  whereby  all  of  the  original  factors  would  have  been 
retained  and  unit  weighted,  was  also  rejected  since  It  would  have  little 
face  validity  or  job-family-specific  relevance.  After  an  evaluation  of  the 
advantages  and  disadvantages  of  these  techniques.  It  was  decided  that  a 
promotion  policy  panel  exercise  would  provide  the  best  technique  to  select 
and  weight  the  final  set  of  job-relevant  promotion  factors. 

General  Process 

The  research  strategy  used  judgment  analysis  or  "policy  capturing,"  a 
statistical  technique  developed  at  the  Manpower  and  Personnel  Division.  The 
procedure  has  been  used  In  several  previous  studies  (Black,  1973;  Gott, 
1974;  Koplyay,  Albert,  &  Black,  1976).  Basically,  the  technique  consisted 

of  convening  a  series  of  Job-famlly-speclflc  experimental  promotion  boards 
(the  Promotion  Policy  Data  Panels,  or  PPDPs)  to  perform  two  tasks:  (a) 
select  a  list  of  6  to  10  promotion  factors  for  use  In  ranking  employees  In 
that  particular  job  family  and  (b)  rank-order  (sort)  a  randomly  selected 
sample  of  promotion  records  for  employees  In  that  job  family  according  to 
the  employees'  promotablllty.  Then  the  ranking  policy  of  the  boards  was 
mathematically  determined  by  weighting  the  factors  such  that  the  sum  of  each 


factor  rating  Multiplied  by  Its  weight  resulted  In  a  score  that  replicated 
the  panel's  rankings. 

III.  PROCEDURE 

Panel  Member  selection 

To  ensure  the  reliability  and  validity  of  the  development  of  promotion 
factor  weights,  a  great  deal  of  attention  was  directed  toward  selection  of 
Individuals  to  serve  as  panel  members  In  the  development  of 
job-fail ly-speclf  1c  policy  equations  and  resulting  factor  weights*  The 
decisions  and  judgments  which  had  to  be  made  In  developing  the  equations  and 
factor  weights  required  the  selection  and  participation  of  a  balanced  mix  of 
highly  competent*  experienced  job  family  representatives.  A  final  list  of 
criteria  for  selecting  the  panel  members  for  each  job  family  was  jointly 

developed  by  a  working  group  of  AFHRL  scientists  and  Office  of  Civilian 

Personnel  Operations  (OCPO)  specialists.  Listed  In  order  of  Importance,  the 
criteria  were  as  follows: 

1*  Panel  Size.  It  was  decided  that  each  panel  was  to  be  composed  of  a 
maximum  of  12  members.  A  panel  size  of  12  members  was  selected  as  the 
optimal  balance  between  convenience  and  representation.  In  this  exercise. 
It  was  vitally  Important  to  represent  fully  the  views  of  many  subgroups* 
according  to  job  series,  supervlsory/nonsupervlsory  status,  major  command, 
minority  group,  and  sex  category  background.  It  was  thought  that  a  group 
size  of  less  than  12  would  prevent  adequate  representation  of  various 
employee  subgroups.  On  the  other  hand,  group  membership  of  more  than  12 
would  be  both  expensive  to  bring  In  on  temporary  duty  to  participate  In  the 

panel  process  and  unwieldy  to  handle  In  necessary  panel  activities. 

2.  Experience.  Each  panel  member  was  to  have  had  at  least  3  years 
experience  In  a  specific  job  series.  A  3-year  experience  requirement  was 
consistent  with  the  minimum  experience  qualification  required  for  various 
personnel  classlflcatlon/quallflcatlon  actions. 

3.  Job  Series  Representation.  It  was  necessary  to  ensure  adequate 
representation  of  the  major  job  series  In  each  job  family.  Therefore,  each 
job  series  was  given  one  panel  member  for  every  8.3  percent  represented  In 
the  jbb  family's  total  population.  (The  value  of  8.3  percent  was  used  as 
the  representation  criterion  since  It  equals  1/12  of  the  family's  population 
and  Is  equivalent  to  one  panel  member  on  the  12-person  panel.)  All  series 
with  less  than  8.3  percent  of  the  family's  total  population  were  pooled,  and 
the  remaining  panel  members  were  randomly  selected  from  this  pool.  This 
technique  allowed  for  both  a  certain  degree  of  control  over  the  series  which 
were  represented  and  a  chance  for  each  series  to  be  represented. 

4.  Grade-Level  Representation.  Panel  members  were  drawn  from  only  the 
upper  half  of  the  grade  levels  within  their  job  series.  The  standard  of  the 
upper  grade  levels  In  each  series,  rather  than  each  family,  was  used.  In 
order  to  give  all  job  series  In  the  family  the  opportunity  to  be  represented 
on  the  panels.  Furthermore,  at  least  one  panel  member  was  selected  from 
each  of  the  top  three  grades  In  the  job  family.  This  criterion  was  designed 
to  ensure  sufficient  expertise  and  job-series  experience  for  the  panel 
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members  to  make  promotion  decisions  about  their  job  family  and  to  ensure 
that  each  panel  would  have  the  expertise  to  make  promotion  decisions  about 
the  upper  grade  levels. 

5.  Hlnorlty/Sex  Group  Representation.  The  three  major  ethnic  groups 
(Black,  Hispanic  Surnamed,  and  Nonmlnorlty/Others)  and  the  two  sex 
categories  were  to  be  equally  represented  on  each  panel.  Thus,  two  members 
of  each  12-»person  panel  were  to  be  selected  from  each  of  the  six  ethnic/sex 
category  combinations.  This  would  result  In  six  males  and  six  females,  with 
ti.ree  of  the  members  Black,  three  Hispanic  Surnamed,  and  three 
Nonmlnorlty/Others.  Equal,  rather  than  proportional,  race  and  sex 
representation  was  used,  on  the  recommendation  of  the  AFHRL  Conference  on 
Human  Appraisal  (Mullins,  1981).  Proportional  representation  would  mean  few 
minorities  would  be  represented  and  their  views  could  be  ‘outvoted,*  whereas 
equal  representation  would  mean  an  equal  voice  for  groups  regardless  of  any 
disproportionate  group  makeup  of  the  job  families.  Equal  representation 
should  provide  the  most  legally  defensible  position.  In  terms  of  equal 
employment  opportunity  considerations.  An  additional  advantage  Is  that 
minority  representation  was  assured  for  many  families  In  which  the  minority 
population  Is  low.  In  the  event  that  the  minority  and  sex  criteria  could 
not  be  met,  the  following  deviations  were  permitted:  (a)  If  both  race  and 
sex  criteria  could  not  be  satisfied,  a  member  who  met  the  mlnorlty^group 
criterion  was  selected;  (b)  If  the  m1nor1ty»group  criterion  alone  could  not 
be  met,  a  member  who  met  the  sex*category  criterion  was  selected;  and  (c)  If 
no  member  could  be  found  who  met  either  criterion,  a  member  was  selected 
randomly,  who  met  the  other  criteria  listed  In  paragraphs  1  to  5. 

(6)  Supervlsory/Hanagerlal  Balance.  Panels  were  selected  such  that  at 
least  half  of  the  members  would  be  supervisors/managers.  However,  for  job 
families  In  which  there  was  a  large  percentage  of  supervisors/managers,  more 
than  half  of  the  panel  members  were  supervisors/managers.  This 
overrepresentation  was  built  Into  the  selection  process,  since 
supervisors/managers  are  the  people  responsible  for  making  policy  and 
evaluating  the  workers.  At  the  same  time,  the  panels  were  not  composed 
entirely  of  supervlsors/mar.agers,  In  order  to  give  the  worker  »*  the 
nonsupervisor  **  some  voice  In  developing  the  new  promotion  system. 

Job  Families  18  through  23  (Wage  Supervisor  and  Wage  Grade  families) 
were  necessarily  treated  differently  In  terms  of  the  supervisory /managerial 
balance.  For  Job  Family  Id  (Wage  Supervisor),  all  of  the  members  were 
supervisors.  For  Job  Families  19  through  23  (Wage  Grade),  half  of  the 
members  selected  from  the  job  family  of  Interest  were  nonsupervisors;  the 
other  half  of  the  members  were  selected  from  the  corresponding  job  series  In 
Job  Family  18.  Therefore,  there  was  a  50/50  supervlsory/nonsupervlsory 
balance  In  Job  Families  19  through  23. 

(7)  local/Nonlocal  Area  Representation.  For  each  panel,  three  members 
were  selected  from  the  local  (Bexar  County)  area,  and  the  other  nine  were 
selected  from  outside  the  local  area.  This  criterion  was  a  compromise 
between  economic  necessity  (l.e.,  amount  of  TOY  costs)  and  proportional 
CONUS-»wide  representation.  Nonlocal  representation  was  random,  to  give  all 
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geographical  areas  an  equal  opportunity  for  representation  depending  on  the 
percentage  of  job  family  representation. 

(8)  Awards.  Panel  Members  who  had  a  history  of  awards  were  chosen,  to 
the  fullest  extent  possible.  It  was  considered  highly  desirable  for  each 
panel  member  to  have  at  least  one  Outstanding  Performance  Report  (OPR), 
Quality  Step  Increase  (QSI),  Sustained  Superior  Perforaance  (SSP)  Award,  or 
Cash  Suggestion  Award.  However,  since  this  criterion  was  the  lowest  in  the 
hierarchy,  and  since  only  a  saall  percentage  of  the  total  population  receive 
Such  awards,  panel  aeabers  with  no  awards  were  selected  when  necessary. 

These  criteria  were  coablned  In  a  computer-based  selection  procedure  In 
the  following  aanner:  First,  the  saaple  of  all  CONUS  Federal  Service 
eaployees  was  reduced  to  a  pool  of  ellglbles,  according  to  the  experience 
and  grade-level  representation  criteria.  A  six-way  distribution  was 
provided  for  each  job  family  (job  series  by  Minority  group  by  sex  category 
by  supervisory/nonsupervlsory  category  by  local/nonlocal  area  group  by 
awards/nonawards  category).  Only  those  cells  having  three  or  More  cases 
that  Met  the  reMalnlng  experience  and  grade-level  criteria  were  selected  In 
order  to  provide  two  alternate  selections  for  each  prlMary  panel  MeMber 
designated.  However,  In  sowe  cases,  It  was  necessary  to  select  froM  a  cell 
with  only  one  or  two  cases  and  select  the  alternate(s)  frow  a  closely  (but 
not  perfectly)  watching  cell.  These  alternates  generally  watched  the 
prlMary  on  all  criteria  but  one  or  two  (e.g.,  sex  group,  ethnic  group,  award 
status).  Once  the  cells  for  sampling  had  been  specified,  the  12  panel 
members  and  alternates  for  each  job  faMlly  were  randOMly  selected  by 
cowputer.  The  final  product  of  the  panel  member  selection  procedure  was  a 
listing  of  36  potential  panel  members  for  each  job  family,  consisting  of  12 
primary  selections  and  a  first  and  second  alternate  for  each  primary. 

At  this  point,  the  names  of  the  PPDP  selectees  (primaries)  were  sent  to 
the  local  Central  Civilian  Personnel  Offices  (CCPOs)  for  confirmation  and 
verification  through  the  employee's  supervisor:  (a)  that  the  employee  was 
available  for  panel  duty,  (b)  that  the  employee  was  willing  to  participate 
and  was  free  from  personal  problems,  and  finally  (c)  that  the  employee  was 
suitable  for  panel  duty.  In  terms  of  experience  and  skill.  The  CCPOs 
replied  with  the  conflrmatlons/decllnatlons,  and  alternate  panel  members 
were  requested  as  necessary.  In  some  cases,  It  was  necessary  to  sample 
additional  alternates  to  fill  a  panel  position;  the  largest  number  of 
additional  alternates  sampled  for  any  one  position  was  10. 

Table  C-1  shows  the  characteristics  of  the  PPDP  members  for  each  job 
family.  It  Is  obvious  fro*  this  table  that  the  criteria  were  not  met  In  a 
number  of  Instances.  Two  major  Impediments  were  encountered  In  finding 
panel  members  meeting  the  exact  criterion  seed flcatlons.  First,  members 
with  certain  necessary  characteristics  either  did  not  exist  or  existed  In 
extremely  small  numbers.  Second,  frequently  the  alternate  selections 
provided  less  than  optimal  matches.  Deviations  required  varied  by  job 
family.  Panel  sizes  varied  from  10  to  12,  due  either  to  last-minute 
cancellations  or  Inability  to  find  an  acceptable  alternate.  Also,  It  was 
sometimes  Impossible  to  obtain  members  from  the  top  three  grades  In  certain 
job  families.  In  other  families,  there  were  very  few  women  or  minority 
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Stgtfflwry  iUtut  S*p*Tiitor  (S)  Sup«rvfs«r  (4)  Supervisor  (4)  Supervisor  (4)  Supervisor  (4) 

XeRSupervIser  (4)  Noetvpervlser  (4)  Xeesupervlsor  (S)  Konsupervlsor  (4)  Konsupervlsor  (4) 


KOTC:  lusters  (n  parentheses  to  the  right  cf  entries  Indicate  the  frequency.  If  sore  than  one, 


I»*r»  1»  MrHtkiut  t*  rlffct  »f  ««trUs  Ut  fr^i«»cyf  If  mr*  tl«»  out. 


lurtvri  In  >tr«Atkct«i  f  tk*  rljkt  if  otrf«s  fckf  fr*qgt*cy,  If  aor«  tk*n  on«, 


group  candidates  who  met  the  desired  experience,  job  series,  and  grade-level 
criteria.  Panel  Membership  froi  the  Bexar  County  area  was  frequently 
overrepresented,  since  local  employees  were  used  as  replacements  for 
Mst-mlnute  cancellations. 

Promotion  Potential  Factors 


The  24  possible  promotion  potential  factors  presented  to  all  job 
families  are  given  In  Table  C-2.  A  full  description  of  the  factors  was 
given  earlier  In  Appendix  B.  A  copy  of  this  list,  along  with  the  Job/Horker 
Characteristics  Ratings  booklet  (which  defines  each  factor  and  Its  rating 
scale),  was  given  to  each  panel  member  for  use  In  the  selection  task. 


Table  C-2.  List  of  Candidate  Factors 


FACTOR 

NUMBER 

SURVEY 

BOOKLET 

NUMBER 

FACTOR  DESCRIPTION 

1 

I  -  1 

OVERALL  JOB  PERFORMANCE  IN  PRESENT  JOB 

2 

I  -  4 

CAPABILITY  TO  SERVE  IN  A  SUPERVISORY  POSITION 

3 

I  *  5 

CAPABILITY  TO  SERVE  IN  A  MANAGERIAL  POSltlON 

4 

III  -  A 

ENERGY 

5 

III  *  B 

SKILL 

6 

III  -  C 

COMPLETION 

7 

III  -  D 

SELF-SUFFICIENCY 

8 

III  -  E 

LEADERSHIP 

9 

III  -  F 

RESPONSIVENESS 

10 

III  -  G 

UNDERSTANDING  OF  OTHERS 

11 

III  *  H 

ORIGINALITY 

12 

III  -  I. 

PRIDE  IN  WORK 

13 

Ill  -  J 

LEARNING  ABILITY 

14 

III  -  K 

SPEAKING  ABILITY 

15 

III  -  L 

WRITING  ABILITY 

16 

IV  -  A 

ARITHMETIC  COMPUTATION 

17 

IV  -  C 

DATA  INTERPRETATION 

18 

IV  *  H 

READING  ABILITY 

19 

VII  -  1 

SPEED  IN  COMPLETING  WORK 

20 

VII  -  2 

PRODUCTIVITY  DURING  WORK  TIME 

21 

VII  *  3 

PERCENTAGE  OF  TIME  ON  THE  JOB  WORKING 

22 

VII  *  4 

INITIATIVE  IN  STARTING  &  COMPLETING  PROJECTS 

23 

III 

SUM  OF  TRAIT  RATINGS 

24 

VII 

SUM  OF  MOTIVATION  MEASURES 
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Preparation  of  Sample  Promotion  Records 

All  promotion  records  presented  to  the  PPDPs  were  drawn  from  those  who 
participated  In  the  CONUS-wlde  field  test  of  the  expanded  Civilian  Potential 
Appraisal  System  In  November  1979  -  May  1980.  Full  details  of  the  sampling 
procedures  and  response  rate  for  this  field  test  are  given  In  Appendix  B.  The 
records  of  all  job  family  employees  who  had  a  complete  Demographic 
Questionnaire-Worker  and  either  a  complete  supervisor's  or  a  complete 
co-worker's  Job/Worker  Characteristics  Rating  were  presented  to  the 
appropriate  panel.  All  records  displayed  to  the  panels  came  from  this  field 
test;  no  artificial  data  were  generated. 

For  purposes  of  the  PPDP  exercise,  the  relevant  data  for  each  employee 
were  presented  on  6.5  by  3.5  Inch  cards  that  displayed  the  relevant  promotion 
factors  representing  one  employee  (see  Figure  C-l).  Each  card  displayed  the 
panel  member's  name  and  number,  deck  number,  job  family  name  and  number,  and 
case  number.  The  promotion  factors  (those  chosen  by  the  panel  during  the 
factor  selection  activity),  along  with  the  employee's  scores  on  these  factors, 
were  given  at  the  bottom  of  the  card.  To  provide  anonymity  to  the  employee 
and  prevent  any  unnecessary  distractions  or  possible  charges  of  rater  bias,  no 
employee  demographic  or  Identifying  Information  such  as  name,  race,  sex,  or 
series,  was  placed  on  the  card. 


PANEL  MEMBER  07  MARIAN  MARTENS  CASE  52 

DECK  NUMBER  07 

JOB  FAMILY  11  BUSINESS  TECHNICAL 

SCORE . FACTOR . 

7  OVERALL  JOB  PERFORMANCE  IN  PRESENT  JOB 

6  SKILL 

9  SELF-SUFFICIENCY 

9  RESPONSIVENESS 

6  PRIDE  IN  WORK 

8  LEARNING  ABILITY 

9  READING  ABILITY 

3  PRODUCTIVITY  DURING  WORK  TIME 

5  INITIATIVE  IN  STARTING  &  COMPLETING  PROJECTS 

16  SUM  OF  TRAIT  RATINGS 


Figure  C-l.  Sample  Card 
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The  total  set  of  cards  for  each  panel  member  was  divided  Into  decks  of  48 
to  100  cards  each,  with  a  target  size  of  70  to  80  cards.  Each  deck 
represented  a  single  grade  level  or  group  of  adjacent  grade  levels,  depending 
on  the  grade^level  combinations  needed  to  compose  decks  of  the  desired  size. 
In  some  Instances,  the  cases  for  a  single  grade  level  were  randomly  split  Into 
twb  or  more  decks  because  of  the  large  number  of  cases  for  that  grade  level. 


In  addition  to  the  basic  data  decks,  each  job  family  panel  member  was 
given  one  reliability  deck  and  (with  the  exception  of  the  five  Wage  Grade 
families)  two  to  five  supervisory  decks.  The  reliability  deck  was  a  duplicate 
of  one  of  the  nonsupervlsory  data  decks  and  was  Included  to  determine  each 
panel  member's  consistency  In  applying  a  particular  promotion  policy  to 
different  data  decks.  This  reliability  deck  was  selected  from  the  largest 
graded evel  grouping  In  each  family;  thus,  It  was  usually  one  of  the  decks 
near  the  middle  of  the  grade*]eve1  range.  The  supervisory  decks  also 
duplicated  certain  decks  that  were  used  In  the  nonsupervlsory  set;  however, 
the  cards  In  the  supervisory  decks  reflected  the  factors  the  panelists  had 
chosen  during  the  supervisory  portion  of  the  factor  selection  process.  The 
number  of  supervisory  decks  reflected  the  proportion  of  supervisors  In  the  job 
family,  and  the  decks  were  chosen  to  reflect  the  grade  levels  with  the  highest 
densities  of  supervisors.  Panel  members  were  not  told  that  the  reliability  or 
supervisory  decks  contained  cases  they  had  seen  previously,  and  no  panel 
member  questioned  whether  or  not  these  were  unique  cases. 


Table  C»3  presents  the  data  deck  compositions  for  the  23  job  families. 
The  number  of  total  case  records  per  family  ranged  from  538  to  1053;  however, 
these  were  not  all  unique  cases,  due  to  the  presence  of  the  reliability  and 
supervisory  decks.  The  Wage  Grade  families  (Job  Families  19  through  23)  had 
no  supervisory  decks,  since  there  are  no  supervisory  positions  In  these 
families.  Job  Family  18  (Wage  Supervisor)  had  nine  basic  decks  (since  all 
employees  In  this  family  are  supervisors)  and  four  "upgrade"  decks,  to  provide 
a  policy  for  promotion  from  a  Wage  Grade  nonsupervisor  to  a  Wage  Supervisor 
position. 
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Table  03.  Data  Deck  Composition 


Administrative  Procedures 


Each  panel  member.  Mho  Mas  confirmed  by  the  CCPO,  received  an  Information 
packet  describing  the  purpose  of  the  PPDP  exercise,  the  techniques  to  be  used 
for  selecting  and  Melghtlng  the  promotion  factors,  the  application  of  the 
final  CPAS  system,  panel  member  selection  procedures,  and  logistical  details. 
The  packet  also  contained  a  listing  of  the  job  series  In  the  member's  job 
family  and  Information  on  the  Randolph  AFB  area,  Mhere  the  panels  Mere  to  be 
assembled. 

All  panels  Mere  assembled  at  Randolph  AFB  on  successive  Meeks  from  22  July 
through  21  September  1980.  Most  panels  met  for  4  Mork  days.  The  first  three 
panels  met  successively  during  the  first  three  Meeks  and  Mere  specially  chosen 
to  represent  anticipated  problems  so  the  lessons  learned  could  be  applied  to 
subsequent  panels.  The  first  panel  to  meet  Mas  Job  Family  10,  Accounting  and 
Finance  Technical.  This  panel  Mas  chosen  because  It  Mas  likely  to  experience 
those  difficulties  common  to  any  job  family  having  a  Mlde  grade  spread  and 
heterogeneity  of  job  series.  Similarly,  Job  Families  1  (Engineering,  Physical 
Science,  and  Mathematical  Professional)  and  14  (Specialized 
Management/Administration),  also  judged  to  be  representative  families.  Mere 
the  next  to  convene.  All  other  panels  folloMed  In  numerical  sequence,  Mlth 
four  panels  meeting  each  subsequent  Meek  of  the  project. 

The  panels  Mere  greeted  en  masse  and  given  a  brief  keynote  address  by  a 
representative  of  the  Office  of  Civilian  Personnel  Operations  (OCPO).  The 
panels  Mere  then  separated  and  turned  over  to  the  respective  panel  leaders. 
(TWo  representatives  of  AFHRL  and  tMO  representatives  of  McFann-Gray  S 
Associates  served  as  panel  leaders  for  each  Meek  In  Mhlch  there  Mere  four 
panels  meeting.)  The  panel  leader  covered  administrative  details  and 
delivered  a  thorough  briefing  concerning  the  purpose  of  the  PPDP,  history  of 
the  CPAS  development,  application  of  the  CPAS  to  PPRS,  job  family  definition, 
panel  member  selection,  and  basics  of  the  PPDP  procedures  (select-sort).  The 
panel  leader  emphasized  that  the  panel's  primary  task  Mas  to  ansMer  the 
question:  "Which  Morkers  Mill  perform  their  jobs  best  If  promoted  to  the  next 
grade  level?"  The  panel  Mas  to  ansMer  this  question  by,  first,  selecting  a 
set  of  relevant  promotion  factors  and,  second,  sorting  a  number  of  case 
records  of  employees  In  the  order  of  their  promotion  potential.  It  Mas 
emphasized  that  the  promotions  concerned  Mere  to  the  next-highest  grade  level. 
In  order  for  the  panels  to  have  a  firm  reference  In  their  promotion 
decisions.  The  distinction  betMeen  current  job  performance  (as  covered  by  the 
Job  Performance  Appraisal  System,  or  JPAS)  and  promotion  potential  Mas  made 
clear,  and  the  panels  Mere  Instructed  to  concentrate  on  the  aspect  of 
promotion  potential.  In  short,  they  Mere  to  Identify  those  factors  that  are 
observable  In  the  current  job  and  are  most  related  to  performance  at  the  next 
higher  level  In  the  job  family. 

An  area  of  some  concern  to  many  panel  members  Mas  hoM  the  decisions  a 
small  group  of  panel  members  Mould  be  generalized  to  a  large  number  of  diverse 
job  series.  Panel  members  Mere  told  that  they  did  not  need  to  knoM  the 
details  of  Job  duties  and  tasks  of  the  other  series  In  their  job  family  beyond 
those  details  knoun  through  their  personal  experience. 


The  select  portion  of  the  PPDP  exercise  was  conducted  on  the  afternoon  of 
the  first  day.  The  panel  leader  gave  a  brief  but  detailed  description  of  the 
rules  for  the  selection  process.  The  panel's  primary  task  was  to  reduce  the 
List  of  Candidate  Factors  to  the  6  to  10  factors  that  would  be  used  In  the 

panel's  own  sorting  exercise  and  In  the  operational  proieotlon  system.  Panels 

were  United  to  10  factors  because  experience  with  policy  capturing  indicates 
that  members  cannot  really  consider  more  than  10  factors  (Black,  1973;  Gott, 
1974;  Koplyay,  Albert,  &  Black,  1976).  Where  a  panel  thought  that  more  than 
10  of  the  factors  were  Important,  they  were  advised  to  construct  composite 
factors  so  that  no  more  than  10  appeared  for  consideration.  Each  panel 

(except  the  Wage  Grade  panels)  was  to  select  two  such  sets  of  factors:  the 

first  for  nonsupervisory  jobs  and  the  second  for  supervisory  jobs.  The 
prescribed  technique  was  to  consider  all  the  factors,  take  an  Initial  vote  on 
the  members'  preferences,  discuss  each  factor,  and  come  to  a  consensus  on 
which  factors  to  use.  The  panel  leader  went  over  each  group  of  Items  In  the 
List  of  Candidate  Factors  (see  Table  C-2)  and  showed  the  panel  members  how  to 
refer  from  the  list  to  the  corresponding  Items  In  the  Job/Worker 
Characteristics  Rating  booklet.  The  leader  also  discussed  Factor  23  (Sum  of 
Trait  Factors)  and  Factor  24  (Sum  of  Motivation  Measures),  emphasizing  that 
Factor  23  might  be  composed  of  any  or  all  of  Factors  4  through  15,  and  Factor 
24  might  be  composed  of  any  or  all  of  Factors  19  through  22.  The  panel  was 
told  that  they  could  select  the  factors  to  make  up  each  composite.  However, 
neither  of  these  two  composite  measures  could  Include  any  Items  that  had  been 
selected  as  Individual  trait/motivation  factors. 

The  panels  were  given  2  hours  to  select  the  set  of  nonsupervisory  factors 
and  1  hour  to  select  the  supervisory  factors.  Throughout  the  discussion,  the 
panel  leader  remained  In  the  conference  room  to  answer  questions  and  provide 
direction  to  the  groups.  The  panel  leader  also  had  a  series  of  computer 
printouts  showing  the  frequency  distribution,  Intercorrelations,  and  Index  of 
potential  adverse  Impact  for  the  candidate  factors.  The  leader  would 
occasionally  use  this  information  to  advise  the  group  on  any  Item  that  showed 
undesirable  statistical  characteristics.  However,  for  the  most  part,  the 
panel  mmbers  were  given  a  free  rein  In  the  selection  process.  As  soon  as 
each  of  factors  was  selected,  the  list  was  communicated  to  the  AFHRL 
computer  programmers  who  began  overnight  preparation  of  the  data  decks  for  use 
by  the  panels  In  the  sort  procedure  the  following  day. 

The  sort  portion  of  the  exercise  began  on  the  second  morning  and  continued 
until  the  panel  members  completed  work  on  their  Individual  decks.  The  panel 
leader  gave  a  brief,  detailed  description  of  the  sort  rules.  The  task  for 
each  panel  member  was  to  rank-order  each  deck  from  the  most-promotable  worker 
to  the  least-promotable  worker.  The  panel  members  were  Instructed  to  work 
Individually  and  avoid  any  discussion  about  which  factors  they  preferred, 
which  workers  should  be  ranked  high,  etc.  Panelists  were  also  Informed  of 
helpful  techniques  for  working  through  the  decks.  It  was  emphasized  that  the 
panelists  should  be  consistent  within  a  grade  level  and  try  to  apply  the  same 
policy  to  all  cases  at  that  level;  however,  they  could  change  their  policy 
from  grade  level  to  grade  level,  If  they  thought  It  appropriate. 


At  the  start  of  the  sort  process,  the  panel  leader  displayed  (a)  the 
minimum  and  maximum  values  for  each  factor,  and  (b)  the  data  deck 
compositions,  including  deck  number,  number  of  cases,  grade  level(s),  and 
supervisory/nonsupervisory  status,  for  each  deck.  The  panel  leader  remained 
in  the  conference  room  with  the  panel  at  all  times  for  immediate  availability 
and  to  ensure  that  the  decks  were  passed  out  and  collected  in  an  orderly 
manner.  The  panel  members  worked  independently,  at  their  own  rates, 
rank-ordering  each  deck.  For  each  deck,  the  cards  were  arranged  by  individual 
panel  members  in  descending  order  of  promotability,  from  the  most  promotable 
worker  to  the  least  promotable  worker.  Each  panel  member  worked  with  only  one 
deck  at  a  time  and  examined  the  decks  in  numerical  sequence  beginning  with 
Deck  1.  The  panel  leader  recorded  the  starting  and  finishing  times  for  each 
member  and  briefly  scanned  the  decks  for  obvious  discrepancies  (e.g.,  a  deck 
ranked  in  reverse  order,  out  of  proper  numerical  sequence,  or  only  partially 
completed).  If  necessary,  the  decks  were  returned  for  reaccompl ishment.  The 
average  time  spent  on  each  deck  was  approximately  45  minutes,  and  most  panel 
members  finished  the  sort  process  by  the  end  of  the  third  workday  or  the 
middle  of  the  fourth  workday.  A  great  deal  of  variation  In  Individual  working 
time  per  deck  was  noted.  An  Individual  average  of  25  minutes  per  deck  was 
found  to  be  the  fastest  working  time,  while  115  minutes  was  the  slowest. 

Quality  Control  and  Data  Reduction  of  Panel  Rankings 

Twice  each  day  during  the  sort  portion  of  the  exercise,  the  completed 
decks  were  collected  from  the  panel  leaders  and  carried  to  the  AFHRL  Technical 
Services  Division  for  immediate  keypunching  of  the  ranking  data.  A  total  of 
three  keypunched  cards  contained  the  essential  information  on  each  deck: 
panel  member  identification  number,  deck  number,  job  family  identification 
number,  and  case  identification  numbers  of  each  of  the  rank-order  positions 
within  a  single  deck.  Keypunchers  also  provided  backup  editing  functions, 
such  as  checking  for  missing  cards,  spotting  misplaced  cards,  etc. 
Approximately  75*  of  the  ranked  decks  were  keypunched,  briefly  edited,  and 
corrected  (If  necessary)  before  the  panel  members  left  Randolph  AFB .  In  three 
instances,  deck  errors  were  spotted  after  the  panel  members  had  left.  These 
decks  were  then  mailed  out  to  the  appropriate  panel  members,  following  a 
telephone  contact,  and  returned  within  2  weeks. 

All  of  the  edited  ranking  data  were  merged  with  the  display  data  for  each 
deck  and  each  board  member,  and  "criterion  ranking  value*  was  generated  for 
each  record.  This  criterion  value  was  generated  by  converting  the  ranks  to 
percentile  scores,  so  that  the  cases  with  the  highest  rankings  (l.e.,  lowest 
numerical  values)  received  the  highest  percentile  scores.  This  procedure 
allowed  rank-ordered  decks  of  different  sizes  to  be  directly  compared  and 
avoided  an  Inevitable  string  of  negative  correlations. 

IV.  RESULTS  OF  PANEL  DATA  ANALYSES 
Promotion  Factor  Selection 


The  majority  of  the  24  candidate  promotion  factors  were  used  by  one  policy 
panel  or  another,  as  either  a  primary  factor  or  part  of  a  composite  factor 
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{ 1  .e. ,  Factor  23  or  24).  Only  Factor  16,  Arithmetic  Computation,  was  not  used 
by  any  panel.  Factor  2,  Supervisory  Capability,  and  Factor  3,  Managerial 
Capability,  were  used  frequently  for  the  supervisory  selections  but  only  once 
for  the  non  supervisory  selections.  With  the  exception  of  these  three  factors, 
all  other  factors  were  used  frequently  for  both  the  supervisory  and 
nonsupervisory  selections. 

The  number  of  factors  used  per  job  family  was  also  quite  high.  For  the 
nonsupervisory  selections,  the  median  number  of  factors  used  (as  either  a 
primary  or  as  part  of  the  composite)  was  15,  with  a  range  of  7  through  20. 
For  the  supervisory  selections,  the  median  was  14,  with  a  range  of  8  through 
22. 

These  findings  were  quite  Important  in  the  final  development  of  the  rating 
form  which  was  to  be  used  In  the  operational  CPAS  system.  A  single,  universal 
rating  form  —  one  that  would  apply  to  all  promotion-eligible  candidates  In 
the  Air  Force  --  was  highly  desirable.  Although  the  difference  In  the  number 
of  factors  used  by  each  job  family  would  favor  a  separate  form  for  each  job 
family  and  for  each  supervlsory/nonsupervlsory  category,  the  high  Incidence  of 
crossflow  between  different  job  families  would  make  the  costs  and  logistics  of 
using  numerous  forms  almost  prohibitive.  Evidence  of  the  high  crossflow  In 
the  current  civilian  personnel  force  was  empirically  demonstrated  by  a  study 
conducted  by  the  Office  of  Civilian  Personnel  Operations.  They  showed  that 
more  job  vacancies  are  filled  by  persons  working  In  another  job  family  than 
are  filled  by  persons  promoted  within  a  job  family. 

From  a  practical  or  economical  standpoint,  the  development  and  use  of  a 
single  form  for  all  job  families  would  mean  that  all  employees  would  have  a 
current  rating  on  all  CPAS  factors;  thus,  a  promotablllty  ranking  score  (CPAS 
score)  could  be  generated  for  any  employee  for  any  job  family.  Such  a  system 
would  allow  an  employee  to  be  considered  for  any  job  whenever  the  occasion 
arose  without  the  need  for  completing  another  rating  form.  On  the  other  hand, 
even  though  a  single  form  Is  desirable.  It  does  have  some  disadvantages.  A 
large  number  of  rating  factors  will  have  the  tendency  to  decrease  the  amount 
of  attention  the  rater  gives  to  each  rating,  thereby  Increasing  the 
possibility  of  halo  effect.  Consequently,  the  value  of  the  ratings  on  the 
"key"  promotion  factors  for  any  given  job  family  may  be  diluted.  These 
disadvantages  to  the  development  of  a  universal  rating  form  had  to  be 
considered  In  the  final  development  of  the  operational  CPAS  form. 

Many  of  these  disadvantages  to  a  single  rating  form  were  eliminated  as 
subsequent  analyses  were  accomplished  in  the  development  of  the  weighted 
algorithms.  For  example,  those  factors  found  to  be  consistently  zero-weighted 
In  the  policy  equations  were  eliminated,  reducing  the  total  number  of  factors 
appearing  on  the  final  form. 

Identification  of  actual  promotion  factors  selected  by  job  family  category 
has  been  removed  from  this  paper  In  compliance  with  the  Directorate  of 
Civilian  Personnel's  decision  that  the  specific  job  family  factors  would  be 
considered  "Privileged  Information." 


Promotion  Weighting  Systems  Developed 


Development  of  the  operational  CPAS  promotion  factor  weighting  systems  or 
algorithms  Involved  four  basic  actions:  (a)  evaluation  of  the  consistency  with 
which  panel  members  ranked  the  promotion  records  and  removal  from  further 
consideration  of  any  member  who  did  not  show  1ntra*rater  consistency,  (b) 
evaluation  of  the  degree  to  which  a  panel  member's  policy  could  be  replicated 
mathematically  and  remove  any  member  from  further  consideration  whose  policy 
could  not  be  replicated,  (c)  Identification  of  the  minimum  numbers  of 
grade*1eve1  and  supervisory  and  nonsupervlsory  within  job  family  policies 
necessary  for  accurate  rankings,  and  (d)  reduction  of  the  resulting  policies 
or  algorithms  to  the  simplest  form  possible  for  operational  use  without 
sacrificing  accuracy. 


Consistency  Analysis.  One  of  the  Initial  concerns  In  the  analysis  of  the 
policy* capturing  data  Is  whether  each  panel  member  maintained  an  acceptable 
level  of  consistency  throughout  the  rank*order1ng  task.  Specifically,  If 
during  the  CPAS  exercises  a  panel  member's  policy  shifted  across  several  decks 
pertaining  to  the  same  grade  level  category,  the  resulting  policy  equation 
(based  on  all  the  decks  that  member  ranked)  would  be  Inconsistent.  As  a 
result,  the  validity  of  this  policy  equation  would  be  questionable,  and  Its 
use  In  developing  the  CPAS  would  be  undesirable.  To  check  panel  member 
consistency,  two  Identical  decks,  about  which  the  panel  members  had  no  prior 
knowledge,  were  Included  among  the  set  of  decks  ranked  by  each  panel  member. 
The  consistency  of  the  rankings  assigned  to  cases  In  the  duplicate  decks  was 
checked  for  each  panel  member  by  computing  the  Pearson  product*moment 
correlation  coefficient  for  the  two  sets  of  rankings.  Table  C*4  summarizes 
the  results  of  the  consistency  analysis  for  all  23  job  families,  with  an 
Interval  frequency  distribution  of  the  resulting  correlation  coefficients  by 
job  family.  For  example,  all  of  the  correlation  coefficients  that  resulted 
from  the  11  Job  Family  3  panel  members  fell  In  the  Interval  .95  through 
1.00. 1  While  all  the  correlations  represented  In  Table  C*4  are 
statistically  significant.  In  a  consistency  analysis,  statistical  significance 
Is  not  the  exclusive  criterion  for  determining  an  acceptable  level  of 
consistency.  The  general  practice  Is  to  screen  out  from  consensus  policy 
development  any  panel  members  whose  duplicate  deck  correlations  deviate 
greatly  from  1.0  and  are  substantially  different  from  those  of  the  other  panel 


^he  possible  range  of  values  for  a  Pearson  correlation  coefficient  is 
*1.0  to  1.0  with  *1.0  Indicating  a  perfect  negative  relation,  .0  Indicating  no 
relation,  and  1.0  Indicating  a  perfect  positive  relation. 
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members.  Although  setting  a  specific  minimum  cutoff  correlation  Is  an 
arbitrary  decision,  past  studies  conducted  by  the  AFHRL  development  team  have 
typically  used  a  correlation  of  .80  as  the  cutoff  for  1ntra*rater 
consistency.  The  same  cutoff  was  used  for  this  study.  For  example.  In  Table 
C’4  the  data  for  Job  Family  20  Indicate  that  one  of  the  panel  members  had  a 
correlation  of  less  than  .80  (actual  correlation  was  .68)  while  the  other  10 
members  on  the  panel  had  correlations  of  .90  or  more.  Consequently,  the  data 
for  the  panel  member  In  question  were  not  considered  to  be  satisfactorily 
consistent  and  were  not  used  In  the  subsequent  analysis  for  consensus  policy 
development.  None  of  the  other  coefficients  represented  In  Table  C*4  were 
considered  to  be  unsatisfactory.  In  fact,  96X  of  them  were  In  the  highly 
satisfactory  range  of  .90  through  1.00,  with  the  remaining  correlations 
falling  no  lower  than  .80. 


Identification  of  Consensus  Panel  Members.  The  objective  of  this  phase 
was  to  determine  If  a  single  equation  could  effectively  be  used  to  represent 
the  rank*order1ng  policies  of  all  the  members  of  a  panel  and  to  provide  that 
equation  for  operational  use.  If  the  panel  equation  differed  from  that  of  an 
Individual  panel  member  or  members,  the  reason  for  the  deviation  must  be 
explored  and  resolved.  Since,  for  most  job  families,  the  panels  provided  data 
for  both  a  nonsupervlsory  and  a  supervisory  policy,  this  Investigation  was 
carried  out  twice  for  each  job  family.  The  procedure  consisted  of,  first, 
developing  a  separate  nonsupervlsory  (and  supervisory)  policy  equation  for 
each  panel  member,  and  then  Inputting  each  respective  set  of  policy  equations 
(nonsupervlsory  and  supervisory)  to  a  hierarchical  grouping  analysis  to 
determine  which  panel  members  (hopefully  all)  had  similar  policies. 


Table  C-4.  Consistency  Analysis  as  Reflected  by  Correlations 
Between  Duplicate  Deck  Rankings 


Job 

Number 

Correlation 

of  Panel  Members  Having 

Coefficients  in  the  Interval 

Family 

less  than 

.80  .80-. 84  .85-.  89 

.90-. 94 

.95-1.00 

1 

0 

1 

0 

0 

10 

2 

0 

0 

0 

2 

8 

3 

0 

0 

0 

0 

11 

4 

0 

0 

0 

2 

10 

5 

0 

1 

1 

0 

10 

6 

0 

0 

0 

0 

11 

7 

0 

0 

0 

0 

12 

8 

0 

0 

0 

1 

11 

9 

0 

0 

0 

1 

9 

10 

0 

0 

0 

0 

11 

11 

0 

0 

0 

1 

10 

12 

0 

1 

0 

0 

10 

13 

0 

0 

0 

1 

10 

14 

0 

0 

2 

0 

9 

15 

0 

0 

0 

0 

12 

16 

0 

0 

0 

1 

10 

17 

0 

0 

0 

0 

10 

18 

0 

0 

0 

0 

10 

19 

0 

0 

0 

2 

9 

20 

la 

0 

0 

3 

7 

21 

0 

0 

0 

2 

9 

22 

0 

0 

2 

1 

9 

23 

0 

0 

0 

1 

11 

aThe 

value  of  this 

correlation 

is  .68.  The 

data  for 

this  panel 

member  were  not  used  for  CPAS  development. 
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Development  of  Individual  panel  member  equations  was  accomplished  using 
regression  analysis.  A  description  of  regression  analysis  as  applied  In 
this  study  can  be  found  In  Bottenberg  and  Hard  (1963).  In  this  application 
of  regression  analysis,  the  objective  Is  to  develop  a  system  of  weights  for 
the  given  set  of  promotion  factors  (or  Independent  variables).  The  weights, 
when  applied  In  a  weighted  sum  with  the  respective  promotion  factor  scores 
for  a  given  employee,  should  approximate  or  predict  the  ranking  (or 
dependent  variable  value)  assigned  to  the  employee  by  a  given  panel  member. 


The  dependent  or  criterion  value  for  the  CPAS  regression  analyses 
consisted  of  the  set  of  percentile  ranks  derived  separately  by  deck  from  the 
panel  member’s  rank^orderlngs.2  To  avoid  negative  correlations  between 
the  promotion  factors  and  the  percentile  rank  criterion,  records  with  the 
highest  rankings  (l.e.,  lowest  numerical  values)  received  the  highest 
percentile  scores.  The  set  of  Independent  or  predictor  variables  used  In  a 
given  regression  analysis  consisted  of  the  set  of  variables  derived  by 
multiplying  each  promotion  factor  by  a  set  of  binary  (0,  1)  variables 
corresponding  to  the  grade»level  categories  Involved.  In  each  analysis, 
there  were  a  x  b  predictors,  where  a  »  the  number  of  promotion  factors  and  b 
«  the  number  of  pertinent  grade»level  categories.3  This  "promotion  factor 
by  grade  level"  regression  model  was  necessitated  by  the  fact  that  the  panel 
members  were  allowed  to  change  their  policies  for  different  grade«1eve1 
categories.  However,  this  regression  model  was  used  only  for  this  phase  of 
the  policy^capturlng  analysis,  with  subsequent  regressions  In  later  phases 
Involving  only  one  grade^level  category  at  a  time. 


2Percent11e  ranks  were  used  as  the  criterion  values  In  lieu  of  the 
actual  rank’order  values  assigned  by  the  panel  members  since  deck  sizes 
varied.  For  example,  since  the  promotion  factor  scores  for  a  case  ranked 
50th  In  a  deck  of  50  records  probably  do  not  reflect  the  same  level  of 
promotablllty  as  those  of  a  case  ranked  50th  In  a  deck  of  90  record:.-  the 
regression  criterion  values  for  two  such  cases  should  not  be  the  same.  An 
acceptable  alternative  to  the  actual  rank^orderlng  was  the  use  of  percentile 
ranks. 

3The  values  for  an  employee  on  the  predictor  variables,  Plj,  for  this 
model  were  defined  as  follows:  Pij  ■  score  on  promotion  factor  1  If  the 
employee  Is  In  grade^level  category  j;  0,  otherwise  for  1  ■  1,  2,  3,  .  .  ., 
a  and  j  *  1,  2,  3,  .  .  .,  b. 
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In  reviewing  the  results  of  the  regression  analyses  that  produced  the 
Individual  panel  member  policy  equations,  the  primary  concern  was  the 
predictive  accuracy  of  each  equation  (i.e. ,  how  well  did  the  policy  equation 
replicate  a  panel  member's  policy).  The  Measure  of  predictive  accuracy  of  a 
regression  equation  Is  the  squared  Multiple  correlation  coefficient  (R2  or 
coefficient  of  Multiple  deterMination),  and  It  Indicates  the  proportion  of 
the  total  variation  about  the  criterion  Mean  explained  by  the  regression 
Model  or  equation.  In  general,  the  predictive  accuracy  of  a  policy  equation 
Is  a  function  of  the  adequacy  of  the  regression  model  (I.e.,  the  set  of 
predictor  variables)  and  the  Internal  consistency  of  the  panel  member  who 
provided  the  rank^orderlngs.  Based  on  past  experience,  the  AFHRL  developers 
set  an  arbitrary  R2  value  of  .81  as  the  minimum  acceptable  multiple 
correlation  for  a  panel  Member's  measure  of  Internal  consistency.  Values 
below  .81  Indicate  that  20  percent  or  more  of  the  variability  In  ranking 
assigned  by  that  panel  Member  Is  not  consistent  with  the  Information 
available  In  Making  those  rankings;  hence,  the  member  Is  not  using  the 
Information  consistently.  Thus,  In  the  process  of  reviewing  the  predictive 
accuracies,  low  r2s  were  carefully  scrutinized  to  determine  whether  they 
were  the  result  of  a  generally  Inadequate  regression  model  or  were  the 
result  of  an  Inconsistent  panel  Member.  The  R2s  which  resulted  for  the  23 
sets  of  nonsupervlsory  policy  equations  developed  in  this  phase  are  reported 
In  Table  C»5,  which  displays  an  Interval  frequency  distribution  of  the  R2s 
by  job  family.  For  example,  for  Job  Family  1,  it  Is  shown  that  10  of  the  11 
Job  Family  1  panel  Members  had  nonsupervlsory  policy  equations  with 
predictive  efficiencies  In  the  Interval  .90  through  .93.  This  Indicates 
that  these  10  policy  equations  contain  90X  to  93<  of  the  Information 
necessary  to  make  perfect  predictions  of  the  panel  member  rankings.  All  but 
one  of  the  Rzs  displayed  In  Table  C®5  fall  In  an  acceptable  range  of 
values  (.81  through  .97).  The  unsatisfactory  value,  .66,  resulted  for  one 
of  the  12  panel  members  for  Job  Family  22.  Since  the  R2s  for  the 
remaining  Job  Family  22  members  are  In  the  Interval  .86  to  .97  and  Indicate 
no  general  problem  with  the  adequacy  of  the  regression  model  for  Job  Family 
22,  It  was  concluded  that  the  low  R2  value  was  the  result  of  Inconsistent 
ranking  procedures  employed  by  the  panel  member  In  question.  Consequently, 
the  data  for  this  panel  member  were  not  used  In  the  development  of  the  Job 
Family  22  nonsupervlsory  CPAS. 


Table  C-S.  Predictive  Efficiency  of  Panel  Member  Nonsupervlsory 

Policy  Equations 


Job 

Family 

Number  of  Panel  Members  Having  Policy  Equations 
with  Predictive  Efficiencies  (R2s}  In  the  Interval 

less  than  .81 

.81-. 85 

.86-. 89 

.90-. 93 

94-. 97 

1 

0 

0 

1 

10 

0 

2 

0 

0 

5 

5 

0 

3 

0 

1 

6 

4 

0 

4 

0 

0 

0 

4 

8 

5 

0 

1 

2 

9 

0 

6 

0 

0 

0 

11 

0 

7 

0 

0 

3 

9 

0 

8 

0 

0 

0 

6 

6 

9 

a 

0 

0 

8 

2 

10 

0 

0 

1 

2 

8 

11 

0 

1 

1 

9 

0 

12 

0 

0 

1 

9 

1 

13 

0 

0 

10 

1 

0 

14 

0 

0 

1 

10 

0 

15 

0 

0 

0 

7 

5 

16 

0 

0 

1 

3 

7 

17 

0 

0 

6 

4 

0 

18 

0 

0 

0 

5 

5 

19 

0 

0 

1 

6 

4 

20 

0 

0 

3a 

7 

1 

21 

0 

0 

1 

1 

9 

22 

lb 

0 

2 

3 

6 

23 

- 5nT7" 

0 

1 

0 

3 

8 

a0ne  of  these  panel  members  was  not  used  for  CPAS  development  as  a 
result  of  having  a  .68  correlation  between  duplicate  deck  rankings  (see 
Table  C-4). 


bThe  value  of  this  R2  Is  .66.  The  data  for  this  panel  member  were 
not  used  for  CPAS  development. 


139 


Information  on  the  predictive  efficiencies  of  the  supervisory  policy 
equations  developed  In  this  phase  Is  summarized  in  Table  C? 6.  The 
Information  given  In  Table  C->6  Is  analogous  to  that  provided  In  Table  C*5, 
with  the  exception  that  data  are  displayed  for  job  families  1  through  18 
only,  since  a  supervisory  CPAS  was  needed  only  for  those  job  families.  All 
of  the  P.2  values  reflected  In  Table  C»6  fall  In  a  highly  satisfactory 
range,  .85  to  .98.  As  a  result,  these  supervisory  policy  equations 
(represented  in  Table  C->6)  were  used  In  the  development  of  the  supervisory 
CPAS. 


As  previously  indicated,  the  next  step  In  this  phase  of  the  analysis  was 
to  determine  If,  and  how,  the  Individual  panel  member  policies  can  be 
combined  to  form  a  single  panel  policy.  The  process  was  to  Input  each 

respective  set  of  member  policy  equations  (nonsupervlsory  and  supervisory) 
for  a  job  family  to  a  separate  hierarchical  grouping  analysis.  The  results 
of  such  an  analysis  provide  the  basis  for  determining  how  to  combine  the 
•separate  panel  member  policies  Into  a  single  policy  which  best  represents 
the  consensus  opinion  of  the  panel.  The  hierarchical  grouping  analysis 
begins  with  the  separate  policy  equations  of  the  panel  members  and 
Iteratively  combines  two  equations  or  two  groups  of  equations  present  at  the 
previous  Iteration.  The  process  continues  until  all  the  policy  equations 

are  combined  Into  a  single  group.  The  two  systems  that  are  combined  at  a 
given  stage  of  the  process  are  those  that  are  the  most  similar  to  one 

another  at  that  point.  They  are  selected  on  the  basis  of  minimum  loss  In 

overall  predictive  accuracy  when  a  compromise  or  joint  equation  Is 
substituted  for  the  separate  systems.  The  overall  predictive  accuracy  Is  a 
maximum  at  the  start  of  the  grouping  process  when  separate  policy  equations 
are  employed  for  each  panel  member.  At  subsequent  Iterations,  the 
predictive  efficiency  decreases  by  successively  larger  amounts  when 
compromise  policies  are  substituted  for  the  separate  policies  being 
combined.  An  appreciable  decrease  In  overall  predictive  accuracy  Is  an 
Indication  that  the  policies  being  combined  are  dissimilar  and  signals  the 
stage  of  the  analysis  which  Identifies  the  largest,  most  homogeneous 

groupings  of  the  panel  members.  A  decision  point  for  how  much  loss  In 

predictive  accuracy  Is  excessive  Is  usually  an  arbitrary  decision  made  by 

the  research  scientist.  For  purposes  of  this  analysis,  no  arbitrary  a 

priori  cutoff  was  set.  An  R2  of  less  than  .75  would,  however,  have  caused 
concern  that  the  predictive  accuracy  of  a  grouping  had  lost  sufficient 
precision  to  warrant  corrective  action.  A  drop  of  .10  In  any  one  grouping 
stage  would  also  have  been  cause  for  concern.  Typically,  In  a  hierarchical 
grouping  analysis,  there  Is  a  large  subset  of  panel  members  found  to  have 
similar  policies,  and  It  Is  this  group  that  Is  used  In  subsequent  analyses 
for  the  development  of  the  "consensus*  policy  equation. 


HO 


£ 

is 


developed. 


The  results  for  the  41*  hierarchical  grouping  analyses  perforated  In 
this  phase  of  CPAS  development  are  summarized  In  Table  C->7.  The  Information 
displayed  for  each  job  family  consists  of  the  overall  predictive 
efficiencies  which  resulted  at  the  Initial  and  final  stages  of  each  grouping 
analysis  (nonsupervlsory  and  supervisory).  For  comparison  purposes,  the 
magnitude  of  the  difference  between  the  Initial  and  final  R2s  Is  also 
given.  For  example.  In  the  grouping  analysis  for  Job  Family  1  the  Initial 
R2  is  .91,  whereas  the  predictive  accuracy  at  the  final  stage  of  the 
grouping  analysis  Is  .88,  a  decrease  of  only  .03.  That  Is,  the  accuracy  In 
predicting  the  nonsupervlsory  rankings  provided  by  the  Job  Family  1  panel 
drops  by  only  .03  when  one  joint  policy  equation  Is  substituted  for  the  11 
Individual  policy  equations  of  the  Job  Family  1  panel  members.  The  outcome 
of  the  grouping  analysis  for  the  supervisory  policy  equations  of  Job  Family 
1  Is  similar,  with  Initial  and  final  stage  predictive  efficiencies  of  .92 
and  .87,  respectively,  which  represents  a  drop  of  .05  In  overall  predictive 
accuracy.  The  results  for  Job  Family  1  are  typical  of  the  results  for  the 
remaining  22  job  families  whose  nonsupervlsory  policy  equation  groupings 
resulted  In  predictive  accuracy  losses  ranging  from  .02  to  .06  (average  loss 
*  .04)  and  whose  groupings  achieved  final  stage  R2s  In  the  range  of  .84  to 
.69.  Similarly,  In  the  supervisory  equation  grouping  results  shown  in  Table 
C^7,  the  R2  drops  range  from  .03  to  .07  (average  loss  =  .04)  with  final 
stage  R2s  In  the  range  of  .87  to  .93.  Such  losses  In  predictive  accuracy 
were  considered  by  the  AFHRL  developers  to  be  minimal  and  acceptable  for 
this  type  of  analysis. 

Although  the  standard  procedure  In  most  pollcy?captur1ng  studies  Is  to 
select  the  largest,  most  homogeneous  grouping  of  policy  makers  present  at 
the  stage  prior  to  an  appreciable  drop  (l.e.,  1.0  or  greater)  In  the  overall 
predictive  accuracy,  the  results  of  the  41  grouping  analyses  In  the  present 
study  necessitated  a  modification  to  that  practice.  It  was  concluded  that 
all  panel  member  policy  equations  should  be  used  1,i  "consensus*  policy 
development  since  none  of  the  drops  In  predictive  accuracy  observed  In  this 
study  were  of  sufficient  magnitude  (from  a  practical  standpoint)  to  warrant 
exclusion  of  any  panel  members.  A  review  of  the  largest  single  drop  In 
overall  predictive  accuracy  within  each  of  the  41  grouping  analyses  revealed 
that  90X  (or  37  of  41)  were  of  the  magnitude  .02  or  less,  with  the  single 
largest  drop  being  .07. 


*There  were  23  nonsupervlsory  policy  equation  grouping  analyses  and 
18  supervisory  policy  equation  grouping  analyses;  a  total  of  41  grouping 
analyses. 


Table  C-7.  Initial  and  Final  Stage  Overall  Predictive  Efficiencies 
Resulting  from  Hierarchical  Groupings 


5U 


Nonsupervlsory  Policy  Grouping  Supervisory  Policy  Grouping 


Job 

Family 

Initial 

Stage  R2 

Final 

Stage  R2 

Decrease 

In  R2 

Initial  Final 

Stage  R2  Stage  R2 

Decrease 

In  R2 

1 

.91 

.88 

.03 

.92 

.87 

.05 

2 

.89 

.87 

.02 

.92 

.89 

.03 

3 

.89 

.84 

.05 

.95 

.92 

.03 

4 

.94 

.91 

.03 

.96 

.89 

.07 

5 

.91 

.86 

.05 

.92 

.88 

.04 

6 

.90 

.86 

.04 

.94 

.91 

.03 

7 

.91 

.88 

.03 

.91 

.87 

.04 

8 

.93 

.89 

.04 

.93 

.89 

.04 

9 

.93 

.89 

.04 

.96 

.93 

.03 

10 

.94 

.91 

.03 

.96 

.93 

.03 

11 

.91 

.87 

.04 

.94 

.91 

.03 

12 

.91 

.87 

.04 

.92 

.87 

.05 

13 

.88 

.85 

.03 

.93 

.90 

.03 

14 

.91 

.86 

.03 

.91 

.87 

.04 

15 

.93 

.91 

.02 

.95 

.91 

.04 

16 

.93 

.89 

.04 

.96 

.90 

.06 

17 

.89 

.85 

.04 

.92 

.88 

.04 

18 

.93 

.88 

.05 

.94 

.90 

.04 

19 

.93 

.88 

.05 

'(onappl  Icable 

20 

.91 

.89 

.02 

Nonappllcable 

21 

.94 

.88 

.06 

No nap  pi Icable 

22 

.92 

.89 

.04 

Nonappllcable 

23 

.94 

.89 

.05 

Nonappl  Icable 

Computation  of  Promotion  Policy  Equations,  The  next  task  in  the 
development  of  the  CPAS  weighting  systems  was  the  computation  of  the  policy 
equations  which  actually  would  serve  as  the  source  of  the  weighting  systems 
themselves.  Three  basic  activities  were  performed  in  arriving  at  the  final 
promotion  policy  equations.  First,  the  consensus  policy  equations  were 
computed  for  each  grade»level  category  within  a  job  family.  Second,  the 
separate  grade  category  policies  within  a  job  family  were  compared  to 
determine  their  degree  of  similarity,  and  third,  additional  policy  equations 
were  computed  when  warranted  to  “collapse"  the  similar  promotion  policies 
across  several  grade^level  categories  Into  a  single  policy.  Since  for  most 
job  families  the  panels  provided  data  both  for  a  nonsupervlsory  and  a 
supervisory  policy,  this  series  of  activities  was  carried  out  for  both  types 
of  policies. 

Consensus  policy  equation  development  for  each  grade>level  category 
within  a  job  family  Involved  the  use  of  regression  analysis,  with  the  Input 
to  the  analysis  consisting  of  the  pooled  data  of  the  consensus  panel  members 
within  a  job  family.  The  dependent  (or  criterion)  variable  values  for  the 
regression  analyses  consisted  of  the  set  of  percentile  ranks  derived 
separately  by  deck  from  the  panel  members*  rank^orderlngs;  however,  the  only 
employee  records  used  In  a  given  analysis  were  those  belonging  to  the 
pertinent  grade^level  category.  The  set  of  independent  or  predictor 
variables  used  in  a  given  regression  analysis  consisted  of  the  set  of 
promotion  potential  factors  (nonsupervlsory  or  supervisory)  selected  by  the 
job  family  panel.  The  results  of  the  analyses  which  produced  the 
(consensus)  nonsupervlsory  and  supervisory  grade»level  policy  equations  are 
summarized  In  Tables  C^8  and  C*9,  respectively.  In  each  table,  Information 
Is  provided  by  job  family  Indicating  the  number  of  grade*level  categories 
for  which  policy  equations  were  derived  and  the  magnitude  of  the  predictive 
efficiencies  (R2s)  for  those  policy  equations  In  an  Interval  frequency 
distribution.  As  an  example.  Table  C»8  Indicates  that  separate 
nonsupervlsory  policy  equations  were  derived  for  five  grade^level  categories 
In  Job  Family  1,  all  five  of  which  had  predictive  efficiencies  In  the  range 
of  .86  to  .95.  These  high  predictive  efficiencies  are  an  Indication  that 
the  11  Job  Family  panel  members  were  Indeed  using  similar  policies,  and  that 
a  single  policy  equation  based  on  the  "pooled*  grade^level  rank^orderlngs  of 
the  panel  members  could  satisfactorily  replicate  the  judgments  of  all  11 
Individual  panel  members.  The  data  presented  In  this  table  for  the 
remaining  22  Job  families  reveal  similar  results  with  80%  (81  of  101)  of  the 
R2s  falling  In  the  range  of  .86  to  .95  and  19%  (19  of  101)  falling  In  the 
range  of  .82  to  .85.  The  Information  In  Table  C?9  for  the  supervisory  grade 
level  policy  equatlors  reveals  that  93%  (53  of  57)  of  the  R2s  fall  In  the 
range  of  .86  to  .95  and  7%  (4  of  57)  of  the  R2s  fall  In  the  range  of  .81 
to  .85. 
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Table  fr»8.  Predictive  Accuracy  of  Grade->LeveT  Nonsupervlsory 
Policy  Equations 


Job 

Family 

Categories 

No.  of 

Grade 

Number  of  Grade^Level  Policy  Equations  Having 

Predictive  Accuracies  (R2s)  In  the  Interval 

less  than  .82 

.82  to.85 

.86 

to  .90 

.91  to  .95 

1 

5 

0 

0 

2 

3 

2 

2 

0 

0 

2 

0 

3 

4 

0 

3 

1 

0 

4 

6 

0 

0 

1 

5 

5 

5 

0 

2 

2 

1 

6 

4 

0 

2 

1 

1 

7 

3 

0 

0 

3 

0 

8 

7 

0 

1 

3 

3 

9 

4 

0 

1 

3 

0 

10 

6 

0 

0 

3 

3 

11 

5 

0 

2 

2 

1 

12 

4 

0 

1 

3 

0 

13 

4 

la 

1 

2 

0 

14 

5 

0 

1 

4 

0 

15 

5 

0 

0 

2 

3 

16 

3 

0 

0 

2 

1 

17 

4 

0 

1 

3 

0 

18 

7 

0 

1 

3 

3 

19 

4 

0 

0 

4 

0 

20 

4 

0 

0 

3 

1 

21 

6 

0 

1 

4 

1 

22 

3 

0 

1 

1 

1 

23 

6 

0 

1 

1 

4 

a 

The  value  Is  .79. 
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No.  of 


o?; 

& 

w 

\ 

fcft 

*\  . 


Job 

Family 


Grade 

Categories 


Number  of  Grade->Level  Policy  Equations  Having 
Predictive  Efficiencies  (R2s)  In  the  Interval 


less  than  .84  .84  to  .85  .86  to  .90  .91  to. 95 


& 

9 

3 

0  0 

0 

3 

i*A 

10 

4 

0  0 

0 

4 

»V. 

fp/II 

11 

3 

0  0 

1 

r. 

sV 

12 

3 

0  0 

3 

0 

| 

13 

4 

0  0 

2 

2 

fy 

:•> 

14 

5 

I*  1 

1 

2 

15 

3 

0  0 

1 

2 

Jyt 

16 

2 

0  0 

1 

1 

§1 

17 

2 

0  0 

2 

0 

18 

3 

0  0 

0 

3 

'ii 

b 

• 

19 

Nonappllcable 

b 

ft 

20 

Nonappllcable 

ft 

b 

i\U* 

s*V 

21 

Nonappllcable 

a: 

22 

b 

Nonappllcable 

ill* 

b 

• 

23 

Nonappllcable 

The  value  Is  .81. 
b 

In  Job  families  19  to  23,  only  nonsupervlsory  policies  were 
developed. 
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As  Indicated  previously,  the  next  activity  In  this  phase  was  to  compare  the 
separate  grade  category  policy  equations  within  a  job  family  to  determine 
their  degree  of  similarity.  The  objective  here  was  to  simplify  the  proposed 
CPAS  as  much  as  possible  and  to  Identify  those  grade  category  policy 
equations  within  a  job  family  which  were  highly  correlated  and  would 
rank->order  promotion  ellglbles  In  the  same  sequence  and,  hence,  could  be 
combined.  If  combination  of  grade  category  policy  equations  were  possible. 
It  would  be  desirable  to  Implement  a  single  policy  equation  that  can  do  the 
same  job  as  two  or  more  which  are  highly  similar.  To  carry  out  this 
Investigation  within  a  given  job  family,  "promotion  scores"  were  computed 
for  each  employee  In  the  job  family  sample  using  each  of  the  separate  grade 
category  policy  equations.5  Then,  Pearson  producfc>moment  correlations 
between  the  obtained  promotion  scores  were  computed  for  each  grade  category 
subsample  separately  within  supervisory  or  nonsupervlsory  classifications. 
(Nonsupervlsory  subsample  sizes  ranged  from  48  to  315,  whereas  supervisory 
subsample  sizes  ranged  from  48  to  114.)  While  the  Intercorrelations 
obtained  are  too  numerous  to  report  Individually,  Tables  C?10  and  C?ll 
contain  Interval  frequency  distributions  of  the  correlations  obtained  by  job 
family.  Table  c*10  contains  the  results  for  the  nonsupervlsory  policy 
Intercorrelations,  and  Table  C^ll  contains  the  results  for  the  supervisory 
policy  Intercorrelations.  Both  Tables  C?10  and  C»ll  reflect  a  high  degree 
of  similarity  among  the  separate  grade^level  policy  equations  developed.  Of 
the  420  Intercorrelations  computed  between  the  nonsupervlsory  grade^level 
policies,  98*  are  .970  or  greater,  with  87*  being  .985  or  larger.  The 
smallest  Intercorrelation  reported  In  Table  C»10  Is  .942.  In  Table  C»ll  for 
the  supervisory  grade^level  policies,  98*  of  the  correlations  are  ,970  or 
greater,  with  80*  of  them  being  .985  or  higher.  The  smallest 
Intercorrelation  reported  among  the  138  correlations  In  Table  C»ll  Is  .954. 
In  view  of  the  high  similarity  between  the  grade^level  policy  equations 
observed  generally  across  all  23  job  families,  the  decision  was  made  by  the 
AFHRL  developers  to  use  a  single  nonsupervlsory  policy  equation  and  a  single 
supervisory  policy  equation  for  each  job  family  to  rank^order  promotion 
ellglbles  within  any  grade  level. 

To  "collapse"  the  grade^level  {nonsupervlsory  and  supervisory)  promotion 
policies  for  a  job  family  Into  a  single,  final  policy  equation,  regression 
analysis  was  used  once  more.  As  before,  the  Input  to  the  analysis  consisted 
of  the  pooled  data  of  the  consensus  panel  members  within  the  job  family,  and 
the  dependent  variable  values  for  the  regression  analysis  consisted  of  the 
set  of  percentile  ranks  derived  separately  by  deck  from  the  panel  members* 


5The  promotion  score  for  an  employee  based  on  a  given  grade  category 
policy  equation  Is  the  sum  of  the  scores  derived  by  applying  the  system  of 
regression  weights  to  his/her  scores  on  the  CPAS  factors,  where  the  weights 
are  those  derived  from  the  regression  analysis  of  the  data  for  that  grade 
category. 


Table  (MO.  Similarity  of  Nonsupervlsory  Grade  Category  Policies 
as  Reflected  by  Intercorrelatlons  of  Promotion  Scores 
by  Grade  Category  Subsample 


s 


Job 

Family 

No.  Of 

Correlations 

Computed 

Number  of 

Correlations 

In  the  Interval 

less  than  .955 

.955'».969 

.970*. 984 

.985->.999 

1 

20 

0 

0 

0 

20 

2 

2 

0 

0 

0 

2 

3 

12 

0 

0 

0 

12 

4 

30 

0 

1 

7 

22 

5 

20 

0 

0 

5 

15 

6 

12 

0 

0 

0 

12 

7 

6 

0 

0 

3 

3 

8 

42 

0 

0 

3 

39 

9 

12 

0 

0 

1 

11 

10 

30 

0 

0 

0 

30 

11 

20 

0 

0 

4 

16 

12 

12 

0 

0 

1 

11 

13 

12 

0 

1 

3 

8 

14 

20 

l8 

1 

5 

13 

15 

20 

0 

0 

0 

20 

16 

6 

0 

0 

0 

6 

17 

12 

0 

0 

2 

10 

18 

42 

lb 

3 

13 

25 

19 

12 

0 

0 

0 

12 

20 

12 

0 

0 

0 

12 

21 

30 

0 

0 

1 

29 

22 

6 

0 

0 

0 

6 

23 

30 

0 

0 

0 

30 

The  value  Is  .942. 
b 

The  value  Is  .945. 
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Table  C*11.  Similarity  of  Supervisory  Grade  Category  Policies 
as  Reflected  by  Intercorrelations  of  Promotion  Scores 
by  Grade  Category  Subsample 


Job 

Family 

No.  of 

Correlations 

Computed 

Number  of  Correlations  In  the  Interval 

less  than  .955 

. 9559. 969 

.9709.984 

.9859.999 

1 

6 

0 

0 

0 

6 

2 

2 

0 

0 

0 

2 

3 

6 

0 

0 

0 

6 

4 

20 

0 

0 

6 

14 

5 

6 

0 

0 

2 

4 

6 

6 

0 

0 

1 

5 

7 

2 

0 

0 

0 

2 

8 

12 

0 

0 

2 

10 

9 

6 

0 

0 

0 

6 

10 

12 

0 

0 

0 

12 

11 

6 

0 

0 

0 

6 

12 

6 

0 

0 

3 

3 

13 

12 

0 

0 

0 

12 

14 

20 

la 

1 

7 

11 

15 

6 

0 

0 

0 

6 

15 

2 

0 

0 

0 

2 

17 

2 

0 

0 

0 

2 

18 

6 

0 

0 

4 

2 

19 

Nonappl  Icicle 

20 

Nonappllcable 

21 

Nonappl Icable 

22 

Nonappllcable 

23 

Nonappllcable 

3The  value  Is  .954. 


rank-»orderings.  However,  Instead  of  Uniting  the  sample  cases  to  those  In  a 
given  grade  category,  the  case  data  for  all  grade  categories  In  the  given 
job  family  were  used  In  the  policy  equation  development.  The  results  of  the 
analyses  which  produced  the  final  (collapsed)  nonsupervlsory  and  supervisory 
policy  equations  for  each  Job  fanlly  are  summarized  In  Table  C*12.  The 
Information  provided  consists  of  the  predictive  efficiencies  achieved  by 
each  policy  equation.  For  the  nonsupervlsory  policy  equations,  the  range  of 
R2  values  Is  .80  to  .89,  with  65*  of  the  R2  values  falling  in  the  range 
of  .85  to  .89.  The  supervisory  policy  equations  have  predictive 
efficiencies  ranging  from  .83  to  .92,  with  78*  of  the  R2  values  falling  In 
the  range  of  .85  to  .92. 


While  these  levels  of  predictive  accuracy  represent  moderate  losses  from 
what  was  previously  obtained  with  separate  policy  equations  for  each 
grade»level  category,  the  simplicity  afforded  to  the  proposed  CPAS  as  a 
result  of  collapsing  policies  across  grade  categories  amply  compensates  for 
the  decrease  In  R2  values.  Additionally,  any  reservations  about  the  use 
of  the  collapsed  policy  equation  In  lieu  of  separate  grade?level  policy 
equations  are  Immediately  allayed  after  a  review  of  the  data  In  Tables  C»13 
and  C?14  which  contain,  by  job  family,  the  correlations  between  the  separate 
grade  category  policy  equations  and  the  collapsed  policy  equation.  (Table 
C»13  contains  Information  for  the  nonsupervlsory  policies,  and  Table  C*14 
pertains  to  the  supervisory  policies.)  In  each  table,  the  Information  shown 
for  each  job  family  consists  of  the  number  of  grade»1evel  categories  for 
which  separate  policies  were  developed  and  an  Interval  frequency 
distribution  of  Pearson  product^moment  correlations.  The  correlations 
reported  were  computed  for  each  grade  category  subsample  separately  (sample 
size  ranges  are  the  same  as  for  Tables  C?10  and  C-»ll )  between  the  final 
policy  equation  promotion  scores  and  the  scores  based  on  the  given 
subsample's  own  grade^level  policy  equation.  For  example.  Table  C?13 
reports  that  each  of  the  five  grade  category  subsamples  comprising  the  Job 
Family  1  sample  yields  a  value  In  the  range  of  .995  to  .999  when  correlating 
Its  respective  grade»level  policy  with  the  final  policy  equation  for  Job 
Family  1.  Both  Tables  C»13  and  C»14  reflect  high  correlations  between  the 
collapsed  policy  equations  and  their  separate  grade»level  counterparts,  with 
94*  and  95*  of  the  correlations,  respectively,  falling  In  the  range  of  .990 
to  .999. 


Table  (M2.  Predictive  Accuracy  of  the  Final  Nonsupervlsory  and 
Supervisory  Policy  Equations 


Job 

Family 

Nonsupervlsory 

Policy 

Supervisory 

Policy 

1 

.85 

.84 

2 

.86 

.88 

3 

.83 

.90 

4 

.86 

.84 

5 

.81 

.86 

6 

.84 

.87 

7 

.84 

.84 

8 

.85 

.88 

9 

.88 

.92 

10 

.89 

.92 

11 

.85 

.88 

12 

.86 

.85 

13 

.83 

.87 

14 

.87 

.85 

15 

.89 

.90 

16 

.88 

.90 

17 

o 

CO 

• 

.87 

18 

.82 

.83 

19 

.86 

a 

Nonappllcable 

20 

.86 

a 

Nonappllcable 

21 

.84 

a 

Nonappllcable 

22 

.88 

a 

Nonappllcable 

23 

.88 

a 

Nonappllcable 

a 

In  job  families  19  to  23,  only  nonsupervlsory 
policies  were  developed. 
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table  C*13.  Intercorrelations  U 
Final  Supervisory  Pol  1c 
Grade  Cal 


Categories  less  than 


The  value  Is  .979 


The  value  Is  .978 
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Table  C*14.  Intercorrelations  Within  Grade  Category  Subsamples  of  the 
Final  Supervisory  Policy  Equations  with  the  Separate 
Grade  Category  Policies 


Job 

No.  of 

Grade 

Number  of  Correlations  In  the 

i  Interval 

Family 

Categories 

less  than  .980 

.980*. 984 

.985*. 989 

.990*. 994 

.995*. 999 

1 

3 

0 

0 

0 

0 

3 

2 

2 

0 

0 

0 

0 

2 

3 

3 

0 

0 

0 

0 

3 

4 

5 

0 

0 

1 

2 

2 

5 

3 

0 

0 

0 

2 

1 

6 

3 

0 

0 

0 

1 

2 

7 

2 

0 

0 

0 

0 

2 

8 

4 

0 

0 

0 

1 

3 

9 

3 

0 

0 

0 

0 

3 

10 

4 

0 

0 

0 

0 

4 

11 

3 

0 

0 

0 

0 

3 

12 

3 

0 

0 

1 

0 

2 

13 

4 

0 

0 

0 

2 

2 

14 

5 

l8 

0 

0 

1 

3 

15 

3 

0 

0 

0 

0 

3 

16 

2 

0 

0 

0 

0 

2 

17 

2 

0 

0 

0 

0 

2 

18 

3 

0 

0 

0 

2 

1 

19 

b 

Nonappl Icable 

20 

b 

Nonappl Icable 

21 

b 

Nonappl Icable 

22 

b 

Nonappl Icable 

23 

b 

Nonappl Icable 

aThe  value  is  .977. 

bIn  job  families  19  to  23,  only  nonsupervlsory  policies  were  developed. 
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Identification  of  Operational  Weighting  Systems,  With  two  exceptions 
(described  In  detail  below),  the  CPAS  weighting  systems  to  be  proposed  for 
operational  use  were  derived  by  "rounding"  the  raw  score  regression  weights 
to  the  nearest  whole  number  using  conventional  rounding  procedures.  Raw 
score  regression  weights  which  were  less  than  .5  (Including  several  negative 
weights)  were  set  to  zero.  When  It  was  noted  that  a  large  percentage  of  the 
nonsupervlsory  and  supervisory  policies  had  rounded  regression  weights  equal 
to  zero  for  proaotlon  factors  17  and  18  (data  interpretation  and  reading 
ability,  respectively),  one  further  step  was  carried  out  In  the  development 
of  the  operational  weights.  The  lapact  of  eliminating  promotion  factors  17 
and  18  was  Investigated  for  those  policy  equations  which  employed  a  nonzero 
weight  for  those  factors.  In  every  case,  proaotlon  scores  based  on  the 
rounded  regression  weight  system,  with  factors  17  and  18  removed,  correlated 
by  at  least  .99  with  proaotlon  scores  generated  by  the  original  system  of 
weights.  Consequently,  the  decision  was  made  to  drop  factors  17  and  18  from 
the  proposed  CPAS. 

One  minor  problem  arose  as  a  result  of  the  decision  to  drop  factors  17 
and  18  from  CPAS.  The  procedure  left  only  four  promotion  factors  In  one  job 
family  supervisory  policy  equation;  a  result  which  would  not  satisfy  the 
Office  of  Civilian  Personnel  Operations  requirement  for  all  policy  equations 
to  utilize  a  minimum  of  five  CPAS  factors.  To  remedy  the  problem,  the 
supervisory  policy  equation  for  the  affected  job  family  was  recomputed  using 
one  final  regression  analysis  wlsn  factors  17  and  18  eliminated  from  the  set 
of  Independent  variables.  The  regression  analysis  was  carried  out  with  the 
expectation  that  one  of  the  factors  which  previously  received  a  raw  weight 
less  than  .5  would  receive  a  larger  weight  with  the  modified  set  of 
Independent  variables.  The  analysis  did  produce  the  desired  results  and 
yielded  a  supervisory  policy  for  the  affected  job  family  with  five  promotion 
factors  In  It. 

To  ensure  that  the  CPAS  weighting  systems  for  operational  use  were  truly 
representative  of  the  judgments  of  the  respective  policy  panels,  two 
additional  analysis  activities  were  accomplished.  The  first  procedure 
entailed  generating  the  ranking  (within  a  deck)  that  the  job  family  sample 
cases  would  receive  on  the  basis  of  their  nonsupervlsory  and  supervisory 
weighting  systems  and  correlating  them  with  the  actual  rankings6  assigned 
by  the  job  family  panel  members.  The  results  of  this  Investigation  are 
reported  In  Table  C»15,  which  contains  the  Pearson  correlation  coefficients 
by  job  family  which  resulted  for  both  the  nonsupervlsory  and  supervisory 
weighting  systems.  For  example,  for  Job  Family  1,  a  value  of  .997  was 
obtained  for  the  correlation  between  the  nonsupervlsory  rankings  assigned  by 
the  Job  Family  1  panel  members  and  the  rankings  generated  by  the  operational 
weights  for  the  nonsupervlsory  Job  Family  1  CPAS.  The  operational  weights 
developed  for  Job  Family  1  supervisory  promotions  performed  exceptionally 
well  too,  with  a  correlation  of  .995  occurring  between  the  panel  judgments 
on  supervisory  promotions  and  the  rankings  generated  by  the  weighting 
system.  All  of  the  correlations  displayed  In  Table  C»!5  Indicate  that  the 
weighting  systems  developed  for  operational  use 

6For  this  Investigation  a  "collective*  rank->order  variable  was 
generated,  based  on  the  sum  of  the  ranks  assigned  by  the  Individual  panel 
members. 
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Table  C^>15.  Intercorrlelatlons  by  Job  Family  Between  Panel  Rankings  and 
the  Rankings  Generated  by  CPAS  Weighting  Systems  for  Operational  Use 


Job 

Family 

Correlation  Between  Panel  Rankings  and  Rankings  Generated  by 

Nonsupervlsory 
Weighting  System 

Supervisory 

Weighting  System 

1 

.997 

.995 

2 

.990 

.993 

3 

.993 

.994 

4 

.992 

.991 

5 

.995 

.993 

6 

.996 

.992 

7 

.989 

.984 

8 

.994 

.992 

9 

.995 

.995 

10 

.997 

.996 

11 

.992 

.994 

12 

.989 

.990 

13 

.995 

.993 

14 

.985 

.991 

15 

.996 

.996 

16 

.995 

.995 

17 

.994 

.994 

18 

.992 

.995 

19 

.995 

Nonappllcable 

20 

.994 

Nonappllcable 

21 

.992 

Nonappllcable 

22 

.991 

Nonappllcable 

23 

.996 

Nonappllcable 
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can  simulate  the  decisions  of  the  policy  panels  extremely  well,  with  90*  (or 
37  of  41)  of  them  falling  In  the  range  of  .990  and  .997  and  the  remaining 
102  falling  no  lower  than  .984. 

In  the  second  analysis  activity  to  check  the  precision  of  the 
operational  form  of  the  CPAS  weighting  systems,  the  two  sets  of  rankings 
generated  In  the  first  activity  were  compared  on  a  case*by»case  basis  to 
determine  whether  the  top  five  employees  who  were  Identified  within  each 
deck  by  the  panel  members  were  also  among,  the  top  five  employees  identified 
by  the  corresponding  operational  weighting  system7.  This  analysis  was 
carried  out,  both  for  the  nonsupervisory  and  supervisory  rankings;  and  the 
results  are  displayed  by  job  family  In  Table  C»16.  The  Information 
displayed  consists  of  the  number  of  data  decks  (nonsupervisory  and 
supervisory)  for  which  the  comparison  Is  reported,  along  with  the  number  of 
decks  for  which  there  was  1002,  802,  602,  and  402  agreement  (on  the  top  five 
employees)  In  the  two  sets  of  rankings.  Among  the  23  nonsupervisory 
weighting  systems,  two  (l.e.,  the  systems  for  Job  Families  2  and  19)  were  In 
1002  agreement  with  the  panel  rankings  In  all  comparisons  performed,  and  16 
systems  maintained  an  overall  minimum  level  of  agreement  of  802,  with 
Incidences  of  1002  agreement  occurring  In  at  least  half  of  the  deck?rank1ng 
comparisons  performed.  Of  the  five  weighting  systems  which  had  an 
occurrence  of  602  agreement  In  a  deck»rank1ng  comparison,  all  but  one  system 
attained  1002  agreement  in  at  least  half  of  the  deck»rank1ng  comparisons 
performed.  Among  the  18  supervisory  weighting  systems,  six  (l.e.,  the 
systems  for  job  families  1,  2,  9,  10,  11,  and  16)  were  in  1002  agreement 
with  the  panel  602  agreement,  but  did  not  otherwise  appear  to  differ 
seriously  from  the  rankings  In  all  comparisons  performed,  and  nine  systems 
maintained  an  overall  minimum  level  of  agreement  of  80  percent.  In  all  but 
one  of  the  nine  systems.  Incidences  of  1002  agreement  occurred  In  at  least 
half  of  the  deck?rank1ng  comparisons  performed.  Two  of  the  supervisory 
weighting  systems  (l.e.,  the  systems  for  Job  Families  3  and  6)  had  an 
occurrence  of  panel  rankings  since  they  each  achieved  1002  agreement  In 
another  comparison.  However,  the  low  rates  of  agreement  which  occurred  for 
Job  Family  7  (l.e.,  402  and  802)  raised  some  degree  of  concern  and  led  to 
more  analysis  of  the  data  to  determine  whether  remedial  action  could  be 
taken. 


'The  comparison  Is  Important  because  as  the  last  ranking  process  In 
Identifying  promotion  ellglbles,  only  the  top  five  candidates  are  frequently 
placed  on  the  certificate  of  ellglbles  which  Is  sent  to  the  selecting 
supervisor. 
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Table  fr>16.  Comparisons  of  Promotion  Data  Panel  Rankings  and  Rankings 
Derived  from  the  CPAS  Weighting  Systems® 


Nonsupervlsory  Weighting 
Systems  Results 

Supervisory  Weighting 
Systems  Results 

Job 

No.  of 
Comparisons 

No.  of  Comparisons 
with  Indicated 
Percent  Agreement 

No.  of 
Comparisons 

No.  of  Comparisons 
with  Indicated 
Percent  Agreement 

Faml ly 

Performed 

60S 

80* 

100* 

Performed 

40* 

60* 

80*  100* 

1 

9 

1 

2 

6 

3 

0 

0 

0 

3 

2 

4 

0 

0 

4 

2 

0 

0 

0 

2 

3 

8 

0 

1 

7 

3 

0 

1 

1 

1 

4 

8 

0 

3 

5 

S 

0 

0 

1 

4 

5 

7 

1 

2 

4 

3 

0 

0 

1 

2 

6 

8 

0 

1 

7 

3 

0 

1 

0 

2 

7 

5 

1 

2 

2 

2 

1 

0 

1 

0 

8 

9 

1 

2 

6 

4 

0 

0 

3 

1 

9 

8 

0 

1 

7 

3 

0 

0 

0 

3 

10 

7 

0 

2 

5 

4 

0 

0 

0 

4 

11 

10 

0 

3 

7 

3 

0 

0 

0 

3 

12 

6 

0 

2 

4 

3 

0 

0 

1 

2 

13 

8 

0 

2 

6 

4 

0 

0 

1 

3 

14 

7 

1 

2 

4 

5 

0 

0 

1 

4 

15 

9 

0 

1 

8 

3 

0 

0 

1 

2 

16 

8 

0 

2 

6 

2 

0 

0 

0 

2 

17 

10 

0 

2 

8 

2 

0 

0 

1 

1 

18 

8 

0 

1 

7 

4 

0 

0 

2 

2 

19 

10 

0 

0 

10 

b 

Nonappllcable 

20 

7 

0 

3 

4 

b 

Nonappllcable 

21 

8 

0 

4 

4 

b 

Nonappllcable 

22 

6 

0 

1 

5 

b 

Nonappllcable 

23 

7 

0 

2 

5 

b 

Nonappllcable 

Total 

177 

5 

41 

131 

58 

1 

2 

14 

41 

a 

Information 

reported 

In  this  table 

Is  based  on 

comparison 

of  the 

top 

five  cases  within  each  set  of  rankings. 

bIn  job  families  19  to  23,  only  nonsupervlsory  policies  were  developed. 


a 

1 

i 


\ 


■•v 
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An  Investigation  of  the  data  used  for  Job  Family  7  supervisory  policy 
development  revealed  eight  employees  within  a  single  deck  who  were  tied  at 
the  maximum  score  on  all  the  factors  In  the  operational  form  of  the  Job 
Family  7  supervisory  weighting  system.  However,  the  eight  subjects  had  a 
varl'ety  of  score  combinations  oh  three  additional  Job  Family  7  supervisory 
promotion  factors  which  subsequently  became  zero?we1ght>.d  during  the 
development  of  the  operational  weights.  In  the  same  data  deck,  there  were 
three  additional  employees  who  scored  the  maximum  on  all  selected  promotion 
factors  except  one,  with  their  performance  on  that  factor  being  only  one  or 
two  points  below  the  maximum.  These  three  cases  were  ranked  higher  by  the 
panel  members  than  six  of  the  eight  employees  who  got  maximum  promotion 
scores  with  the  operational  weighting  system,  thus  causing  only  40% 
agreement  between  the  rankings  of  the  operational  weighting  system  and  those 
of  the  Job  Family  7  panel  members. 

To  determine  whether  the  precision  of  the  Job  Family  7  supervisory 
weighting  system  could  be  Improved,  rank^orderlngs  of  the  supervisory  cases 
were  reaccomplished  using  three  revised  weighting  systems.  The  revised 
weighting  systems  consisted  of  the  original  operational  system  with  one  of 
the  previously  zero^welghted  factors  again  Incorporated.  One  of  the  revised 
systems  yielded  100i  agreement  (on  the  top  five  employees)  for  each  of  the 
two  supervisory  deck  comparisons  and  was  subsequently  recommended  for  the 
operational  Job  Family  7  CPAS. 

The  operational  algorithms  generate  an  employee's  CPAS  score  by  applying 
the  appropriate  job  family  Integer  weighting  systems  (nonsupervlsory  or 
supervisory)  to  the  employee's  score  on  the  respective  CPAS  promotion 
factors.  For  example,  the  CPAS  score  for  an  employee  In  a  job  family  with 
five  non^zero  weighted  factors  would  be: 

CPAS  Score  -  (Fwl  x  Frl)  +  (Fw2  x  Fr2)  +  (Fw3  X  Fr3) 

+  (FW4  x  Fr4>  +  (Fw5  x  FrS) 

where  FW4  represents  the  Integer  weight  for  a  factor  and 

Fr  represents  the  factor  rating  given  the  employee  In  the  CPAS  rating 

The  factor  weights  are  whole  Integers  and  range  from  1  to  10  for  the 
nonsupervlsory  and  1  to  11  for  the  supervisory  weighting  systems.  Tables 
C*17  and  C»18  display  the  possible  point  ranges  for  the  nonsupervlsory  and 
supervisory  CPAS  weighting  system  along  with  the  CPAS  score  ranges,  means, 
and  standard  deviations  for  the  job  family  sample  cases  used  In  the 
development  of  the  respective  weighting  systems.  For  example,  In  Table  C»17 
the  minimum  and  maximum  possible  scores  using  the  Job  Family  1  CPAS 
nonsupervlsory  weighting  system  are  20  and  172,  respectively,  while  the 
range  of  scores  observed  In  the  Job  Family  1  sample  (used  In  the 
nonsupervlsory  policy  development)  Is  37  to  172,  with  a  mean  score  of  131.3 
and  a  standard  deviation  of  26.9. 
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Weighting  Systems  Comparisons 


Analysis  of  the  panel  data  and  the  development  of  the  CPAS  weighting 
systems  were  concluded  with  a  final  activity  to  determine  the  similarity 
among  the  41  promotion  policies  which  resulted.  There  were  two  types  of 
comparisons  performed. 

To  determine  the  degree  of  agreement  among  the  23  nonsupervlsory 
weighting  systems,  Pearson  correlations  were  computed.  Input  to  the 
analysis  consisted  of  promotion  scores  computed  for  each  case  In  a  job 
family  sample  using  each  of  the  23  CPAS  nonsupervlsory  weighting  systems. 
For  each  job  family  sample,  correlations  were  computed  between  the  promotion 
scores  based  on  the  given  job  family's  nonsupervlsory  weighting  system  and 
the  scores  generated  with  the  22  other  nonsupervlsory  weighting  systems. 
While  the  23  sets  of  22  correlations  obtained  are  too  numerous  to  report 
Individually,  Table  C->19  contains  an  Interval  frequency  distribution  of  the 
correlations  obtained  by  job  family.  For  example,  the  22  correlations 
obtained  for  Job  Family  1  fall  Into  the  three  Intervals,  .95  to  .96,  .97  to 
.98,  and  .99  to  1.00,  with  frequencies  of  1,  9,  and  12,  respectively. 
Indicating  that  the  Job  Family  1  CPAS  nonsupervlsory  weighting  system 
correlates  by  .95  or  higher  with  each  of  the  weighting  systems  for  Job 
Families  2  to  23  (for  cases  In  the  Job  Family  1  sample).  In  general,  the 
data  In  Table  C19  Indicate  a  considerable  amount  of  similarity  between  the 
23  CPAS  nonsupervlsory  weighting  systems,  with  54*  (272  of  506)  of  the 
correlations  being  .99  or  higher.  While  no  specific  patterns  for  grouping 
the  23  job  family  policies  are  Immediately  apparent  when  reviewing  the  506 
correlations.  It  appears  that  several  groupings  could  be  developed  with 
marginal  losses  In  precision  for  the  purpose  of  simplifying  the 
nonsupervlsory  CPAS. 

The  process  for  comparing  the  18  supervisory  CPAS  weighting  systems  was 
analogous  to  the  process  used  for  the  nonsupervlsory  systems.  For  each  job 
family  sample,  correlations  were  computed  between  promotion  scores  based  on 
the  given  job  family's  supervisory  weighting  system  and  the  scores  generated 
with  the  17  other  supervisory  weighting  systems.  The  results  of  the 
correlation  computations  are  summarized  In  Table  C»20,  where  an  Interval 
frequency  distribution  of  the  18  sets  of  17  correlations  Is  provided  by  job 
family.  In  general,  the  data  In  Table  C»20  reflect  considerably  less 
agreement  between  supervisory  weighting  systems  than  was  observed  for 
nonsupervlsory  systems  In  Table  C->19.  A  decision  was  made  between  the  AFHRL 
developers  and  OCPO  advisors  that  it  was  desirable  to  maintain  separate 
weighting  systems  for  each  job  family.  Of  the  41  resulting  CPAS  weighting 
systems,  only  40  were  delivered  to  USAF/MPK  for  operational  Implementation 
since  there  was  no  operational  requirement  for  a  nonsupervlsory  algorithm 
for  Job  Family  18,  Wage  Supervisor. 

Summary  of  Analyses.  A  summary  of  the  analyses  used  In  developing  the 
operational  CPAS  algorithms  Is  presented  In  Table  C»21.  This  table  provides 
a  brief  description  of  the  objective,  statistical  technique,  and  result(s) 
for  each  of  12  analyses  used  In  the  development  process. 
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Table  C*17.  Possible  Score  Ranges  and  Sample  Statistics  for  the 
Nonsupervlsory  CPAS  Weighting  Systems 


Sample  Statistics 

Job  Possible  Scores  for  Total  Job  Family  Sample 


Family 

Minimum 

Maximum 

Mean 

S.D. 

Min. 

Max. 

Na 

1 

20 

172 

131.3 

26.9 

37 

172 

714 

2 

27 

219 

180.1 

30.4 

68 

219 

320 

3 

21 

173 

134.1 

28.0 

35 

173 

513 

4 

24 

204 

152.8 

32.4 

52 

204 

545 

5 

18 

162 

123.9 

24.6 

54 

162 

460 

6 

18 

158 

119.6 

25.0 

21 

158 

555 

7 

22 

174 

136.7 

27.6 

40 

174 

353 

8 

23 

195 

149.5 

29.9 

50 

195 

650 

9 

22 

190 

138.2 

29.9 

32 

190 

587 

10 

20 

172 

124.9 

30.2 

35 

172 

574 

11 

20 

160 

122.3 

26.1 

40 

160 

700 

12 

25 

209 

158.1 

33.2 

25 

209 

422 

13 

20 

164 

132.4 

24.4 

28 

164 

543 

14 

15 

119 

93.6 

20.2 

25 

119 

439 

15 

19 

163 

124*7 

27.6 

28 

163 

576 

16 

19 

163 

121.7 

27.9 

37 

163 

640 

17 

17 

137 

105.9 

25.2 

19 

137 

726 

18 

20 

180 

137.6 

28.1 

27 

180 

547 

19 

20 

164 

119.1 

27.4 

42 

164 

669 

20 

20 

164 

118.7 

27.2 

20 

164 

488 

21 

24 

176 

126.7 

27.8 

38 

176 

545 

22 

24 

192 

139.9 

31.2 

34 

192 

480 

23 

25 

193 

139.8 

29.3 

45 

193 

565 

*The  sample  size  (N)  reported  represents  the  number  of  job  family 
sample  cases  used  In  the  development  of  the  given  weighting  system. 
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Table  (M8.  Possible  Score  Ranges  and  Sarple  Statistics  for  the 
Supervisory  CPAS  Weighting  Systems 


Job 

Family 

Possible  Scores 

Sample  Statistics 

for  Total  Job  Family  Sample 

Minimum 

Maximum 

Mean 

S.D. 

Min. 

Max. 

N4 

1 

18 

162 

121.0 

26.6 

45 

162 

256 

2 

24 

204 

163.2 

30.9 

56 

204 

167 

3 

15 

135 

95.4 

25.7 

15 

135 

189 

4 

21 

177 

126.5 

29.0 

57 

177 

324 

5 

18 

162 

124.4 

25.2 

47 

162 

206 

6 

19 

167 

117.8 

30.0 

27 

167 

198 

7 

28 

236 

189.0 

37.0 

82 

236 

141 

8 

17 

153 

116.9 

24.9 

39 

153 

278 

9 

20 

180 

120.2 

29.9 

43 

180 

214 

10 

16 

144 

97.4 

27.7 

25 

144 

315 

11 

19 

163 

111.1 

29.2 

29 

163 

229 

12 

17 

153 

109.3 

26.1 

34 

153 

225 

13 

16 

140 

102.4 

25.5 

16 

140 

283 

14 

14 

126 

94.2 

22.6 

22 

126 

315 

15 

20 

172 

122.2 

29.5 

31 

172 

209 

16 

17 

153 

110.8 

26.9 

30 

153 

162 

17 

19 

155 

117.6 

28.1 

33 

155 

114 

18 

22 

198 

150.8 

30.9 

35 

198 

232 

19 

b 

Nonappllcable 

20 

b 

Nonappl Icable 

21 

b 

Nonappllcable 

22 

b 

Nonappllcable 

23 

b 

Nonappllcable 

®The  sample  size  (N)  reported  represents  the  number  of  job  family 
sample  cases  used  In  the  development  of  the  given  weighting  system. 
bln  job  families  19  to  23,  only  nonsupervisory  policies  were  developed. 
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Table  019.  Similarity  of  23  Nonsupervlsory  CPAS  Weighting  Systems  as 


i 


$ 
*1  cj 

§ 

I 


I 


I 


a 

Reflected  by  Intercorrelations  of  Promotion  Scores  by  Job  Family 


Job 

Family 

Number  of  Correlations 

in  the  Interval 

less  than  .95 

.95  ..  .96 

.97  »  .98 

.99  *  100 

1 

0 

1 

9 

12 

2 

0 

0 

7 

15 

3 

1b 

5 

14 

2 

4 

0 

0 

6 

16 

5 

0 

7 

15 

0 

6 

0 

0 

6 

16 

7 

0 

0 

10 

12 

8 

0 

0 

6 

16 

9 

0 

1 

8 

13 

10 

0 

1 

12 

9 

11 

0 

2 

4 

16 

12 

0 

0 

5 

17 

13 

0 

1 

13 

8 

14 

0 

1 

8 

13 

15 

0 

0 

6 

16 

16 

0 

1 

7 

14 

17 

0 

0 

6 

16 

18 

lb 

2 

14 

5 

19 

0 

2 

5 

15 

20 

0 

1 

10 

11 

21 

3c 

6 

9 

4 

22 

0 

2 

7 

13 

23 

0 

2 

7 

13 

®An  Interval  frequency  distribution  Is  provided  for  the  22 
correlations  obtained  when  correlating  promotion  scores  based  on  a  given  job 
family’s  nonsupervlsory  weighting  system  with  promotion  scores  generated  by 
the  22  other  nonsupervlsory  weighting  systems. 
bThe  correlation  represented  Is  .94. 

CThe  correlations  represented  are  .91,  .93,  and  .94 
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Table  fr>20.  Similarity  of  18  Supervisory  CPAS  Weighting  Systems  as  Reflected 
by  Intercorrelatlons  of  Promotion  Scores  by  Job  Family* 


Job 

Number  of 

Correlations  In 

the  Interval 

Family 

less  than 

.93  .93  » 

.94  .95  *  .96 

00 

• 

* 

N 

Q* 

• 

.99  *  1.00 

1 

0 

1 

4 

7 

5 

2 

lb 

5 

2 

7 

2 

3 

ld 

3 

2 

5 

6 

4 

0 

2 

4 

8 

3 

5 

0 

0 

0 

9 

8 

6 

0 

2 

2 

5 

8 

7 

4C 

3 

2 

7 

1 

8 

0 

0 

1 

10 

6 

9 

2e 

2 

3 

3 

7 

10 

lb 

3 

1 

5 

7 

11 

le 

3 

4 

4 

5 

12 

0 

0 

1 

11 

5 

13 

lc 

3 

2 

6 

5 

14 

0 

0 

7 

4 

6 

15 

0 

0 

1 

6 

10 

16 

0 

6 

1 

8 

2 

17 

0 

0 

2 

11 

4 

18 

0 

4 

4 

8 

1 

19 

Nonappllcable* 

20 

Nonappllcable^ 

21 

Nonappl1cablef 

22 

Nonappl1cablef 

23 

Nonapp11cablef 

aAn  Interval  frequency  distribution  Is  provided  for  the  17  correla? 
tlons  obtained  when  correlating  promotion  scores  based  on  a  given  job 
family's  supervisory  weighting  system  with  promotion  scores  generated  by  the 
17  other  supervisory  weighting  systems. 
bThe  correlation  represented  Is  .92. 
cThe  correlatlon(s)  represented  are  no  lower  than  .91. 
dThe  correlation  represented  Is  .90. 
eThe  correlatlon(s)  represented  are  no  lower  than  .89. 
fIn  job  families  19  to  23,  only  nonsupervlsory  policies  .  were 


developed. 
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APPENDIX  D 


UNITED  STATES  AIR  FORCE 
CIVILIAN  POTENTIAL  APPRAIfAL 


*  dZfXh  * 


PART  I  -  PERSONAL  DATA 


EMPLOYEE  ( t.o^(  nnu-.  First,  MitUlle  Initial) 


PERIOD  OP  APPRAISAL 


ORGANIZATION  ASSIGN  EO  TO  (Also  enter  of  it  Co  symbol) 


REASON  FOR  APPRAISAL  (Check  one) 

f  I  annual  (■■']  appraisal  current 


[7]  OTHER  (Specily)  , 


pay  plan  and  job  series 


PART  II  -  OFFICIALS’/EMPLOYEE'S  SIGNATURES 


RATING  OFFICIAL  {Su/xvrihof) 

1>at  t 

NAMC.  GRADE,  outy  title 

signature 

ENOORSING  OFFICIAL  'Option*!) 

OAT  t 

name.  GRAOE.  out>  title 

signature 

EMPLOYEE 

SIGNATURE  (Doe*  not  mdic*te  employee  agreement  or  dlamdreement  with  the  epprttianl) 


„7  1^87  REPLACES  AF  FORMS  2439  THROUGH  AF  FORM  2>(«l  WHICH  ARE  OBSOLLTC.  P ACE  1  Of  4  P AGES 

“  •’*  ■«"'  »"“U%  0“  L  ■  O  W>  L>  .>  v1  *  L*1  -  v  -  ^  L'*  .  •  V'*  I'  *  '  »  *  k  Pk  •  -  v  »  »  «  , 


PART  III  -  RATING  PURPOSE  AND  INSTRUCTIONS 


PURPOSE:  Part  IV  describes  19  behavioral  dimensions.  The  ratings  you  assign  to  each  of  these  dimensions  will  result  in  an  overall  employee  ranking 
that  will  be  used  for  competitive  promotion,  reassignment,  change  to  lower  grade,  and/or  selection  for  training.  While  ratings  are  baaed  on  observable 
behavior  in  the  current  position,  an  overall  rating  indicating  employee  potential  will  be  derived  based  upon  the  requirements  of  the  position  to  be  filled. 
INSTRUCTIONS:  U)  Based  on  your  observations  of  the  employee,  rate  EVERY  behavioral  dimension.  Although  employees  should  receive  a  wide 
range  of  ratings  showing  their  particular  strengths  and  weaknesses,  most  ratings  should  be  In  the  CENTRAL  RANGE.  (2)  Use  the  following  scale  in 
making  the  ratings.  Circle  the  scale  number  in  the  appropriate  box  next  to  each  dimension.  A  rating  of  1,  2,  8,  or  9  requires  justification.  Justifica¬ 
tion  must  be  legible  and  limited  to  the  bracketed  space  provided. 

LOW  RANGE  CENTRAL  RANGE  HIGH  RANGE 


LOW  f^ANGE 

1.  VERY  POOR 

2. '  FAR  BELOW  AVERAGE 

3.  Below  sverage 


4.  Slightly  below  average 

5.  Average 

6.  Slightly  above  average 


PART  iV  -  BEHAVIORAL  DIMENSIONS 


HIGH  RANGE 

7.  Above  average 

8.  FAR  ABOVE  AVERAGE 

9.  OUTSTANDING 


1.  The  employee  Is  energetic  on  the  job;  is  willing  to  exert  effort  accomplishing  tasks. 


2.  The  employee  performs  job-associated  tasks  well,  whether  they  require  physical, 
mechanical,  technical,  professional,  or  managerial  skills;  is  considered  very  skillful 
on  the  job. 


12  3 


4  5  6 


7  8  9 


I  3.  The  employee  follows  through  well;  accomplishes  all  tasks  required  to  complete  a  job 
in  a  timely  manner  on  his/her  own.  1  J  T-^  I  3 


4  5  6 


7  8  9 


4.  The  employee  works  independently  with  little  need  for  additional  supervision  or  help. 


12  3 


4  5  6 


7  8  9 


5.  The  employee  inspires  others  to  action;  accomplishes  goals  by  hfving  a  positive 
influence  on  the  behavior  of  others. 


12  3 


4  5  6 


7  8  9 


6.  The  employee  understands  and  carries  out  oral  or  written  instruction. 


1  2l  3 


4  5  6 


7  8  9 


7.  The  employee  understands  the  behavior  of  fellow  workers,  superiors,  and  sub¬ 
ordinates;  can  "stand  in  other  person's  shoes." 


1  2l  3 


4  15  |6 


7  I  8  _  9 


8.  The  employee  devises  new  solutions  to  problems;  creates  new  methods  and  proce¬ 
dures  for  accomplishing  objectives. 


12  3 


9.  The  employee  takes  pride  in  doing  good  work  and  producing  a  fust-rate  product; 
strives  to  be  best  at  whatever  he'she  does. 


12  3 


4  5  6 


4  5  6 


r  7  ttt 
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PART  IV  -  BEHAVIORAL  DIMENSIONS  (Continued) 


10.  Th-i  employee  picks  Up  neW  Ideas  and  procedures  quickly;  is  easy  to  instruct; 
can  adapt  to  the  demands  of  new  situations. 


11. 

The  employee  explains,  instructs,  and  converses  with  others  in  a  clear  and 
ctive  manner. 

effe 

2 

Jj 

L< 

5 

6 

7 

1 8 

9 

12. 

The  employee  prepares  written  materials  that  are  effective  and  easily  understood. 

T 

2 

:u 

Li_ 

5 

6 

{j_ 

1 8 

9 

□ 

13. 

The  employee's  overall  job  performance  in  his/her  present  job. 

"■ 

2 

Ll 

5 

6 

Ll 

1 8 

9 

r 

14.  The  employee's  ability  to  direct  and  tram  others,  oversee  and  document  work  acti¬ 
vities,  select  and  evaluate  personnel,  implement  management  directives,  or  substitute 
for  absent  supervisor. 

1 

2 

m 

Gl 

5 

6 

C E 

9 

15.  The  employee's  ability  to  implement  Air  Force  directives  and  regulations;  plan,  _________  ___________  r___r___ 

organize  and  monitor  work  projects;  or  represent  the  unit  'hrough  demonstrations  or  j  I  2  I  3  4  I  5  6  I  7  I  8  I  9 

briefings.  - - -  - -  - - — 


ADDITIONAL  INSTRUCTIONS.  Using  the  scale  given,  circle  the  appropriate  fating  on  each  statement  below. _ 

16.  Compared  to  other  individuals  doing  about  the  same  type  of  work,  does  the  employee  complete  his/her  work  (projects,  duties  and  tasks)  faster?  '{ 

Faster 

VERY  MUCH  FASTER 
FAR  FASTER 

□ 

17.  Compared  to  other  individuals  doing  ibout  the  same  type  of  work,  how  is  the  employee’s  work  productivity  during  his/her  work  time? 


1 

FAR  SLOWER 

4 

Slightly  slower 

7 

2 

VERY  MUCH  SLOWER 

5 

The  same  speed 

8 

3 

Slower 

6 

Slightly  faster 

9 

c 


1 

THE  LEAST  PRODUCTIVE 

4 

Slightly  less  productive 

7 

More  productive 

2 

MUCH  LESS  PRODUCTIVE 

5 

The  same  productivity 

8 

MUCH  MORE  PRODUCTIVE 

3 

Less  productive 

6 

Slightly  more  productive 

9 

THE  MOST  PRODUCTIVE 

I] 


18.  Compared  to  other  individuals  doing  about  the  same  type  of  work,  does  the  employee  spend  a  greater  percentage  of  time  on  the  job  working,  as  opposed 
to  socializing,  sitting  idle,  involved  in  personal  affairs,  taking  breaks,  etc.? 


1 

THE  LEAST  TIME  WORKING 

4 

Slightly  less  time  working 

7 

More  time  working 

2 

MUCH  LESS  TIME  WORKING 

5 

The  same  time  working 

8 

MUCH  MORE  TIME  WORKING 

3 

Less  time  working 

6 

Slightly  more  time  working 

9 

THE  MOST  TIME  WORKING 

□ 


19  Compared  to  other  individuals  doing  about  the  same  type  of  work,  does  the  employee  show  more  initiative  in  starting,  carrying  out  and  completing 
projects7  j 

More  initiative  j 

MUCH  MORE  INITIATIVE 
THE  MOST  INITIATIVE 

□  1 


1 

THE  LEAST  INITIATIVE 

4 

Slightly  less  initiative 

7 

2 

FAR  LESS  INITIATIVE 

5 

The  same  initiative 

8 

3 

Less  initiative 

6 

Slightly  more  initiative 

9 
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APPENDIX  E 


CIVILIAN  POTENTIAL  APPRAISAL  SYSTEM  SUPERVISORY  BROCHURE 


'  SECTION  I 

INTRODUCTION 

The  Civilian  Potential  Appraisal  System  (CPAS)  was  designed  to  provide  an  appraisal 
procedure  for  competitive  actions  such  as  promotion,  reassignment,  change  to 
lower  grade,  and  selection  for  training.  Since,  in  many  cases,  the  CPAS  deter¬ 
mines  whether  a  person  is  referred  for  a  job  or  not,  the  CPAS  is  extremely 
important  to  managers,  supervisors,  and  employees.  The  CPAS  was  developed 
based  on  a  thorough  research  and  development  effort.  The  purpose  of  CPAS  is 
to  make  meaningful  distinctions  among  qualified  employees. 

AFR  40-335,  The  Merit  Promotion  Program,  authorizes  CPAS  as  a  ranking  factor 
in  competitive  actions.  Employees  with  a  fully  successful  rating  or  higher 
on  the  Job  Performance  Appraisal  System  (JPAS)  or  the  General  Manager  Appraisal 
System  (GMAS)  will  be  considered  for  placement  using  merit  promotion.  All 
appropriated  fund  employees  must  receive  CPAS  ratings  except  GS-16  through 
GS-18  and  SES. 

Your  ratings  must  be  based  on  observable  behavior  of  the  employee's  current  posi¬ 
tion.  You  will  not  be  rating  an  employee's  potential  on  the  behavioral  dimensions. 
The  employee's  potential  will”  be  determined  by  using  formulas,  these  formulas 
were  developed  based  on  groups  of  Air  Force  civilian  positions  that  possess  simi¬ 
lar  job  requirements.  Twenty-three  job  families  were  identified  which  possess 
similar  characteristics  or  job  requirements.  Subject  matter  experts  from  each  of 
these  23  job  families  participated  in  the  development  of  weights  to  be  assigned 
to  the  various  CPAS  behavioral  dimensions.  These  weights  vary  by  job  family  and 
supervisory/nonsupervisory  characteristics  of  the  position  being  filled.  A  total 
of  40  different  formulas  are  used  in  CPAS  to  predict  employee’s  potential  for  suc¬ 
cess  in  various  positions  being  filled.  Therefore,  since  an  employee  may  be  in 
competition  for  positions  in  a  variety  of  job  families  and  supervisory/nonsupervisory 
positions,  there  is  no  single  score  which  you  can  provide  an  employee  as  a  result 
of  CPAS  ratings. 


SECTION  II 
RATING  EXPLANATIONS 


A  serious  problem  with  previous  appraisal  systems  has  been  inflation  of  the 
ratings  to  the  top  of  the  rating  scale.  When  most  of  your  employees  receive 
the  highest  possible  ratings,  the  appraisal  process  becomes  ineffective.  Your 
ratings  under  CPAS  should  not  be  influenced  by  any  previous  ratings  given. 

It  is  up  to  you  as  a  rater  to  make  CPAS  ratings  meaningful. 

In  Using  the  rating  scale,  you  must  keep  in  mind  that  the  typical  or  "AVERAGE" 
Air  Force  employee  works  hard  and  is  very  proficient  on  the  job.  You  should 
consider  this  behavior  as  what  would  be  expected  of  most  employees.  Be  sure 
to  keep  in  mind  this  idea  of  "AVERAGE"  when  rating  your  employees.  Most  employ¬ 
ees  should  receive  a  wide  range  of  ratings  showing  their  particular  strengths 
and  weaknesses;  most  ratings  should  be  in  the  "CENTRAL  RANGE."  Ratings  of 
1,  2,  8 ,  or  9  require  narrative  explanation  and  approval  by  the  reviewing  offi¬ 
cial. 

The  following  example  is  provided  to  help  you  understand  the  different  rating 
levels.  Behavioral  Dimension  3  is  "The  employee  follows  through  well;  accom¬ 
plishes  all  tasks  required  to  complete  a  job  in  a  timely  manner  on  his/her 
own . " 


An  employee  rated  1  (VERY  POOR)  in  the  LCW  RANGE  would 
have  taken  more  than  the  time  allowed  to  complete  most 
tasks,  and  would  have  almost  always  failed  to  follow  through 
on  his  or  her  own  in  completing  the  task. 

An  employee  rated  5  (AVERAGE)  in  the  CENTRAL  RANGE  would 
have  completed  most  tasks  in  less  than  the  maximum  time 
allowed,  and  almost  always  would  have  followed  through 
well  on  his  or  her  own  in  completing  tasks  in  the  time 
allowed . 

An  employee  rated  9  (OUTSTANDING)  in  the  HIGH  RANGE  would 
have  completed  all  tasks  in  far  less  than  the  maximum  time 
allowed,  and  always  would  lave  followed  through  well  on 
his  or  her  own  in  completing  tasks  in  the  time  allowed. 


CPAS  Behavioral  Dimensions  16-19  are  also  based  upon  a  9-point  rating  scale, 
with  the  points  of  the  scale  grouped  into  a  "LOW  RANGE"  (ratings  1-3),  "CENTRAL 
RANGE"  (ratings  4-6),  and  "HIGH  RANGE"  (ratings  7-9).  Note,  however,  that 
the  definitions  of  the  rating  scales  are  slightly  different  than  the  scale 
used  for  the  first  15  behavioral  dimensions 
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The  following  examples  are  provided  to  help  you  understand  the  different  rating 
levels  applied  in  the  last  four  behavioral  dimensions.  Behavioral  Dimension 
19  states  "Compared  to  other  individuals  doing  about  the  same  type  of  work, 
does  the  employee  show  more  initiative  in  starting,  carrying  out,  and  completing 
proj ects?" 


An  employee  rated  in  the  LCff  RANGE  (1-3)  would  have  demon¬ 
strated  the  least  to  less  initiative  on  the  job  in  completing 
projects  when  compared  to  other  individuals  doing  about 
the  same  type  of  work. 

An  employee  rated  in  the  CENTRAL  RANGE  (4-6)  would  have 
demonstrated  slightly  less  to  slightly  more  initiative 
on  the  job  in  completing  projects  when  compared  to  other 
individuals  doing  about  the  same  type  of  work. 

An  employee  rated  in  the  HIGH  RANGE  (7-9)  would  have  demon¬ 
strated  more  to  the  most  initiative  on  the  job  in  completing 
projects  when  compared  to  other  individuals  doing  about 
the  same  type  of  work. 


When  rendering  ratings,  you  should  remember  that  employees  must  be  rated  within 
the  scope  of  their  jobs.  For  example,  Behavioral  Dimension  15  is  "The  employee's 
ability  to  implement  Air  Force  directives  and  regulations,  plan,  organize  and 
monitor  work  projects,  or  represent  the  unit  through  demonstrations  or  briefings. 


Wage  schedule  employees  might  not  be  concerned  with  many 
Air  Force  regulations  and  directives  but  would  be  quite 
involved  with  organizing  and  monitoring  their  own  work 
projects.  While  they  might  not  present  formal  demonstra¬ 
tions  or  briefings,  they  would  explain  and  demonstrate 
work  procedures  to  other  employees. 

Behavioral  Dimension  12  is  "The  employee  prepares  written  materials  that  are 
effective  and  easily  understood." 


There  are  some  Air  Force  employees  whose  jobs  do  not  require 
preparation  of  written  materials .  However,  all  employees 
are  required,  from  time  to  time,  to  complete  certain  types 
of  written  matter,  such  as  forms,  talley  sheets,  etc. 

You,  as  the  supervisor ,  should  rate  these  employees  on 
their  observable  abilities  to  successfully  complete  a  written 
product.  For  example,  a  Food  Service  Worker  may  have  to 
complete  talley  sheets  of  the  inventory  of  dishes.  This 
is  a  written  product  that  the  employee  must  complete. 

There  may  be  some  instances  in  which  you  may  be  rating 
an  employee  who  cannot  write.  In  these  cases,  you  must 
still  give  the  employee  a  rating  on  the  Behavioral  Dimension. 
The  rating  must  be  in  the  1-9  range.  The  most  appropriate 
rating  in  this  case  would  probably  be  * 1 "  (VERY  POOR). 
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In  most  cases ,  both  of  these  types  of  employees  will  seldom 
be  in  competition  for  positions  in  which  Behavioral  Dimension 
12  is  given  any  significant  weight  in  the  formula  for  the 
position  to  be  filled. 


SECTION  III 

INSTRUCTIONS  FOR  COMPLETION  OF 
CIVILIAN  POTENTIAL  APPRAISAL  FORM,  AF  FORM  1287 

This  section  describes  the  sequence  that  you,  the  rating  official,  will  follow 
in  completing  the  CPA  form. 

Part  I:  Personal  Data 

This  part  contains  pertinent  employee,  position  and  appraisal  information.  If 
a  preprinted  label  does  not  accompany  the  Civilian  Potential  Appraisal  Form, 

AF  Form  1287,  you  must  enter  the  employee's  name,  organization  and  office  symbol, 
social  security  account  number  (SSAN),  current  pay  plan,  job  series  and  grade  in 
the  appropriate  blocks  provided  on  AF  Form  1287.  All  personal  data  should  be 
verified.  Enter  in  the  blocks  provided  in  the  "Period  of  Appraisal"  area  the 
dates  covering  the  period  of  time  for  which  the  employee  is  being  rated.  Check 
the  appropriate  reason  for  giving  the  appraisal  rating  in  the  "Reason  for 
Appraisal"  area.  ANNUAL  -  Check  this  box  if  the  rating  is  required  for  the 
regular  annual  CPAS  rating  cycle.  APPRAISAL  CURRENT  -  Check  this  box  if  there 
is  no  change  in  the  current  rating  on  record  and  it  should  continue  in  effect. 

If  this  is  the  case,  do  not  complete  Part  IV  of  the  appraisal  form.  However, 
you  must  complete  Part  II  as  described  below.  OTHER  -  Check  this  box  if  the 
rating  is  being  given  for  other  than  the  annual  rating  cycle.  You  must  specify 
the  reason  for  completing  the  rating;  for  example,  assignment  of  a  new  Air  Force 
employee  with  no  CPAS  rating  and  who  needs  a  rating  within  90  days  of  assignment. 

Part  III.  Rating  Purposes  and  Instructions 
This  part  of  the  form  should  be  read  prior  to  completing  Part  IV. 

Part  IV:  Behavioral  Dimensions 

Now,  that  you  have  read  the  instructions  and  rating  scale  given  in  Part  III,  com¬ 
plete  Part  IV  in  pencil .  All  19  behavioral  dimensions  (elements)  must  be  rated 
on  each  employee  supervised.  Remember  to  base  your  ratings  on  actual  observations 
of  the  employee.  Although  most  employees  should  receive  a  wide  range  of  ratings 
showing  their  particular  strengths  and  weaknesses,  most  ratings  should  be  in  the 
CENTRAL  RANGE.  Circle  the  rating  scale  number  in  the  appropriate  box  for  each 
behavioral  dimension.  A  rating  of  1,  2,  8  or  9  will  require  written  justification. 
The  justifications  must  be  legible  and  limited  to  the  bracketed  space  provided  on 
the  form.  They  should  be  clear,  brief,  include  specific  job  examples  and  should 
not  merely  repeat  the  behavioral  dimension.  You  may  want  to  write  your  tentative 
justifications  on  bond  paper  in  the  event  of  change  by  the  Reviewing  Official. 

The  following  behavioral  dimensions  are  provided  as  examples  to  aid  you  in  writing 
justifications: 


BEHAVIORAL  DIMENSION: 


Part  II:  Official  s' /Employee's  Signatures 


This  part  contains  the  final  actions  required  to  complete  a  CPAS  rating  after 
you  have  completed  Part  IV  as  described  above. 

HIGHER  LEVEL  REVIEW  -  After  you  (the  Rating  Official)  have  completed  Part  IV 
in  pencil,  submit  the  AF  Form  1287  to  the  Reviewing  Official  (normally  your 
supervisor)  for  review  and  approval  The  Reviewing  Official  must  sign  and 
date  the  form  in  the  appropriate  blocks  provided  if  there  is  agreement  with 
the  tentative  rating  that  you  have  assigned.  If  disagreement  occurs,  efforts 
to  resolve  differences  must  be  made;  however,  the  decision  of  the  Rev ;r ting 
Official  is  final.  ENDORSING  OFFICIAL  -  This  is  an  optional  area  and  additional 
instructions  will  be  issued  if  this  Is  to  be  used.  RATING  OFFICIAL  (SUPERVISOR) 
After  the  reviewing  official  has  reviewed  and  signed  the  CPA  form,  you  must 
finalize  the  numerical  ratings  and  the  narrative  justifications  in  ink  (or 
typewritten)  for  all  entries  in  Part  IV.  You  must  sign  and  date  the  form  in 
the  Rating  Official  Signature  Block  provided.  EMPLOYEE  DISCUSSION  -  The  CPAS 
rating  must  be  shown  and  discussed  with  the  employee.  Employee's  signature 
should  be  obtained.  This  signature  does  not  indicate  employee  agreement  or 
disagreement  with  the  appraisal.  If  employee  refuses  to  sign  the  form,  you 
must  enter  in  the  employee  signature  block  that  the  employee  has  seen  the  rating 
and  enter  the  date  the  discussion  has  occurred. 


SECTION  IV 
RATER  ACCURACY 


The  success  of  the  CPAS  depends  on  you  -  the  rater  -  making  unbiased,  objective 
ratings. 

CPAS  ratings  are  important:  They  provide  you  and  other  supervisors  with  a 
tool  to  distinguish  among  the  qualified  candidates. 

When  rating  your  employees  on  the  19  Behavioral  Dimensions: 

DO 


-  Consider  each  dimension  on  its  own  merit. 

-  Rate  all  persons  on  the  same  behavioral  dimension  at  one  time;  i.e.,  rate 
all  employees  on  Behavioral  Dimension  1  then  everyone  on  Behavioral  Dimen¬ 
sion  2,  etc. 

-  Identify  each  person's  strengths  and  weaknesses  and  ensure  they  are  reflected 
in  your  ratings. 

-  Analyze  your  feelings  and  motivations  regarding  the  ratings. 

-  Have  a  clear  understanding  of  the  terms  used. 

-  Be  observant;  make  a  point  to  observe  the  person  at  work. 

-  Contact  the  previous  supervisor(s)  if  you  have  not  had  sufficient  opportu¬ 
nity  to  observe  the  person  being  rated. 

DON'T 

-  Rate  based  upon  an  overall  impression  of  the  person. 

-  Rate  based  upon  a  recent,  non-typical  event. 

-  Rate  all  employees  in  the  high  or  low  range. 

-  Rate  based  on  the  grade  level  of  the  person  being  rated. 

-  Rate  everyone  in  the  CENTRAL  RANGE  to  avoid  having  to  provide  written 
justification. 

-  Assume  that  a  logical  connection  exists  between  two  or  more  behavioral 
dimensions. 
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SECTION  V 

RATING  EVALUATIONS 

After  you  have  rated  all  of  your  employees,  and  before  you  forward  the  ratings 
to  the  reviewing  official,  you  should  review  your  success  as  a  rater  by  complet¬ 
ing  the  Rating  Distribution  Worksheet  and  the  Rating  Distribution  Chart.  Then 
analyze  the  results  to  ensure  that  you  have  not  committed  common  rating  errors. 

COMPLETE  THE  FOLLOWING  STEPS  FOR  ALL  19  BEHAVIORAL  DIMENSIONS. 


rating 


Step  1:  Using  a  form  like  that  shown  in  Figure  1,  make  a  mark  for  each 
ng  that  you  gave  on  the  19  behavioral  dimensions. 


The  example  below  shows  the  ratings  that  a  supervisor  gave  to  10  employees 
on  Behavioral  Dimension  1. 

LOW  RANGE  CENTRAL  RANGE  HIGH  RANGE 


Behavioral  Dimension  1  022 


1  1  1 


2  1  0 


Step  2;  Total  the  number  of  marks  in  each  of  the  rating  columns.  Figure 
2  shows  a  good  sample  of  this  step. 

Step  3:  Using  a  format  similar  to  Figure  3,  plot  the  total  ratings  for 
each  of  the  9  levels.  Figure  4  shows  a  good  sample  of  this  step. 

Step  4:  Analyze  your  Rating  Distribution  Worksheet  and  Rating  Distribution 
Chart.  Do  they  resemble  Figures  2  asrJ  4? 

Now,  if  you  are  confident  that  your  ratings  are  accurate  and  reflect  the  strengths 
and  weaknesses  of  employees  and  will  be  useful  in  making  meaningful  distinctions 
for  competitive  actions,  you  are  ready  to  forward  your  ratings  to  the  reviewing 
official . 

Review  Process 

If  you  are  also  a  Reviewing  Official,  you  are  responsible  for  (1)  ensuring  that 
subordinate  rating  officials  have  accomplished  their  ratings  in  accordance  with 
the  principles  and  guidance  in  Section  IV  of  this  brochure  and  (2)  resolving 
inconsistencies  among  subordinate  raters.  You  should  use  Rating  Distribution 
Worksheets  and  Rating  Distribution  Charts  for  evaluating  the  CPAS  ratings  given 
to  all  the  employees  in  organizations  that  you  supervise. 
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RATING  DISTRIBUTION  WORKSHEET 


RATING  SCALE  LOW  RANGE  CENTRAL  RANGE  HIGH  RANGE 

1  ?  3  456  7  3  9 


RATING  SCALE 


TOTALS 


Fioure  2 


RATING*  DISTRIBUTION  WORKSHEET 


LOW  RANGE 


CENTRAL  RANGE 
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HIGH  RANGE 
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RATING  DISTRIBUTION  CHART 


10  X  19  (ratings)  =  190  total  ratings  rendered 


